Quarkonium

Form posts using PHP with cURL and Tor
... or posting via anonymous proxy ...

Hi! I'll give an example here of using PHP curl to test website forms over anonymous proxy using Tor.

Tor (originally short for The Onion Router) is a worldwide network of relays used to avoid surveillance and traffic analysis.

Let's assume you have Tor installed and running on the default port 9050. In the example code that follows we first setup the curl options then post some parameters to our form using curl_exec. After examing the response we decide whether to continue with more iterations or not...

          
<?php
  //Contains our test data array ...
  require_once("test_array.php");

  //URL   
  $url = $FORM_URL;

  //Headers
  $headers = array(
    'Host: $FORM_HOST',
    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Referer: $FORM_URL'
  );
    
  //Tor socket address, default port is 9050
  $tor = '127.0.0.1:9050';

  //curl
  $ch = curl_init();
   
  //Set the proxy address
  curl_setopt($ch, CURLOPT_PROXY, $tor);
   
  //Set the proxy type
  curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);

  //The URL of the form we're posting to 
  curl_setopt($ch, CURLOPT_URL, $url);
   
  //Set request headers
  curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
   
  //Prepare for the POST operation
  curl_setopt($ch, CURLOPT_POST, 1);
   
  //Follow any "Location: " header that the server sends
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
   
  //Don't return HTTP headers
  curl_setopt($ch, CURLOPT_HEADER, 0);
   
  //Return the contentof the call
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
  
  //Loop over an array of test data ... 
  foreach($array as $key => $name)
  {
    //Set POST fields, for example ...
    $data = array (
      "email" => $SOME_CONTACT_EMAIL,
      "Address" => $REGISTRATION_ADDRESS,
      "Country" => $COUNTRY,
    );
 
    //traverse our test array (from require_once) and 
    //prepare the data for posting (key1=value1)
    foreach( $data as $key => $value) 
    {
      $post_items[] = $key . '=' . $value;
    }

    //Using implode(), create the post parameter string ...
    $post_string = implode ('&', $post_items);
   
    $post_items = array();

    print_r("$post_string\n");

    curl_setopt($ch, CURLOPT_POSTFIELDS, $post_string);

    $result = curl_exec($ch);

    echo "result $result\n";

    //E.g. test the response for the presence of some field...
    if (strpos($result, $PATTERN) === false) 
    {
      print_r("Found expected pattern $PATTERN in response");
     
      die('OK');
    }

    //Continue test ...
    print_r("Tested fields, continue \n");
  } 
?>

Have fun!


Using the AWS EC2 API in Flex

Welcome! This month I'm going to give a quick tutorial elucidating some basic and very useful AWS EC2 API calls from within Flex. At the end, you can download some example code which you can use as a starting point in building your own AWS API library for Flex! For this tutorial you'll need an AWS account, if you haven't already got one then you can sign up here you can then create and configure an EC2 micro instance using the free usage tier.

In the name of Iron Maiden! Let's begin!

Amazon Web Services provides two types of API for controlling our EC2 instances one is SOAP based the other is REST based, we'll be using the latter in this tutorial

Let's assume we have our EC2 instance set up. Now in your AWS account navigate to your EC2 dashboard. You should see your list of configured EC2 instances. In my test I'm using the server below named 'ec2-server'

Notice the Instance column above, this field contains the instance id that is used to identify our EC2 instance. We'll be using this in the code that follows.

Let's look at how we can programmatically examine the status of any our instances. For this we're going to use the DescribeInstances API call which takes an optional instance id. Supplying the instance id returns information on just that particular instance, while supplying no instance id returns the status of *all* of our instances.

Below is the xml returned in my test set up as a result of the DescribeInstances call ... 

    
<DescribeInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2012-12-01/">
  <requestId>2050b11e-2a9a-419a-b8ae-73249f0a7878</requestId>
  <reservationSet>
    <item>
      <reservationId>r-52ef3136</reservationId>
      <ownerId>393619079936</ownerId>
      <groupSet>
        <item>
          <groupId>sg-b08914d8</groupId>
          <groupName>quick-start-1</groupName>
        </item>
      </groupSet>
      <instancesSet>
        <item>
          <instanceId>i-eaedd492</instanceId>
          <imageId>ami-aecd60c7</imageId>
          <instanceState>
            <code>80</code>
            <name>stopped</name>
          </instanceState>
          <privateDnsName/>
          <dnsName/>
          <reason>User initiated (2013-02-01 18:31:16 GMT)</reason>
          <keyName>test_server</keyName>
          <amiLaunchIndex>0</amiLaunchIndex>
          <productCodes/>
          <instanceType>t1.micro</instanceType>
          <launchTime>2013-02-01T18:24:26.000Z</launchTime>
          <placement>
            <availabilityZone>us-east-1c</availabilityZone>
            <groupName/>
            <tenancy>default</tenancy>
          </placement>
          <kernelId>aki-88aa75e1</kernelId>
          <monitoring>
            <state>disabled</state>
          </monitoring>
          <groupSet>
            <item>
              <groupId>sg-b08914d8</groupId>
              <groupName>quick-start-1</groupName>
            </item>
          </groupSet>
          <stateReason>
            <code>Client.UserInitiatedShutdown</code>
            <message>Client.UserInitiatedShutdown: User initiated shutdown</message>
          </stateReason>
          <architecture>x86_64</architecture>
          <rootDeviceType>ebs</rootDeviceType>
          <rootDeviceName>/dev/sda1</rootDeviceName>
          <blockDeviceMapping>
            <item>
              <deviceName>/dev/sda1</deviceName>
              <ebs>
                <volumeId>vol-08cee669</volumeId>
                <status>attached</status>
                <attachTime>2012-07-25T13:50:11.000Z</attachTime>
                <deleteOnTermination>true</deleteOnTermination>
              </ebs>
            </item>
          </blockDeviceMapping>
          <virtualizationType>paravirtual</virtualizationType>
          <clientToken>ozLCT1343224207944</clientToken>
          <tagSet>
            <item>
              <key>Name</key>
              <value>ec2-server</value>
            </item>
          </tagSet>
          <hypervisor>xen</hypervisor>
          <networkInterfaceSet/>
          <ebsOptimized>false</ebsOptimized>
        </item>
      </instancesSet>
    </item>
  </reservationSet>
</DescribeInstancesResponse>

Notice the instanceState block wtihin the instanceSet showing the instance status.

    
          <instanceState>
            <code>80</code>
            <name>stopped</name>
          </instanceState>

Let's look at the code, our demo library contains an EC2 class which handles the execution of the various API calls using the RESTful interface. In the calling class EC2APIDemo below we instantiate the EC2 class and set up the handler to process the results.

    
public function EC2APIDemo()    
{
  var AWS_ACCESS_ID:String = "YOUR_AWS_ACCESS_KEY";      
  var AWS_SECRET_KEY:String = "YOUR_AWS_SECRET_ACCESS_KEY";      
  var AWS_EC2_SERVER_INSTANCE_ID:String = "i-eaedd492";
 
  var ec2Interface:EC2 = new EC2(EC2.US_EAST_1);
  ec2Interface.setAWSCredentials(AWS_ACCESS_ID,
                                AWS_SECRET_KEY);
        
  var parameterArray:Array = new Array();
  parameterArray.push(new Parameter("InstanceId.1", 
                                     AWS_EC2_SERVER_INSTANCE_ID));
        
  ec2Interface.addEventListener(AWSEvent.RESULT, handler);      
  ec2Interface.executeRequest(EC2.DESCRIBE_INSTANCES, parameterArray);
}

Here's the handler, all we're doing for now is iterating over the ids and the associated state

    
public function handler(event:AWSEvent):void
{
  var xmlData:XML = new XML(event.data);
  trace("handler " + xmlData);
  for each(var item:* in xmlData.*::reservationSet.*::item.*::instancesSet.*::item)
  {
    trace("  item.*::instanceState.*::name " + item.*::instanceState.*::name);
    trace("  item.*::instanceId " + item.*::instanceId);
  }
}

This brings us to our next EC2 API call: Having examined the instance state and finding that it's stopped, we can make use of the StartInstances API calL to get the instance running. As you can see looking at the docs, this expects one or more instance ids so you can start up a whole cluster of them if you want!

    
public function startInstances():void
{
  var parameterArray:Array = new Array();
  parameterArray.push(new Parameter("InstanceId.1", AWS_EC2_SERVER_INSTANCE_ID));

  this.ec2Interface.addEventListener(AWSEvent.RESULT, handlerStartInstances);
  this.ec2Interface.executeRequest(EC2.START_INSTANCES, parameterArray);
}
    
  
    
public function handlerStartInstances(event:AWSEvent):void
{
  var xmlData:XML = new XML(event.data);
  trace("handlerStartInstances " + xmlData);
 
  //Do stuff ... 
  //Error handling ...
}

The response from the StartInstances call as described in the API, results in a list of instance state changes. In this case the state is changed to 'pending' i.e. startup is in progress ...

    
<StartInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2012-12-01/">
  <requestId>ef28014b-4fbc-408f-85b9-3d09b9e655d4</requestId>
  <instancesSet>
    <item>
      <instanceId>i-eaedd492</instanceId>
      <currentState>
        <code>0</code>
        <name>pending</name>
      </currentState>
      <previousState>
        <code>80</code>
        <name>stopped</name>
      </previousState>
    </item>
  </instancesSet>
</StartInstancesResponse>

Taking a look under the hood in the demo Flex EC2 API library we have the method executeRequest. Here we can see the url request constructed using the given EC2 API action e.g. DescribeInstances, StartInstances, StopInstances ... 

    
public function executeRequest(action:String, urlVariablesArr:Array=null, requestMethod:String="POST"):void
{
  this.action = action;
  if(!urlVariablesArr)
  {
    urlVariablesArr = new Array();
  }

  urlVariablesArr.push(new Parameter("Version", "2012-12-01"));
  urlVariablesArr.push(new Parameter("Action", action));
  urlVariablesArr.push(new Parameter("AWSAccessKeyId", awsAccessKey_));
  urlVariablesArr.push(new Parameter("Timestamp", 
                                     AWSDateUtil.generateRequestTimeStamp(new Date())));

  var urlVariables:URLVariables=generateSignature(urlVariablesArr, 
                                                  signatureVersionToUse, 
                                                  requestMethod);
  for each (var item:Parameter in urlVariablesArr)
  {
    urlVariables[item.key]=item.value;
  }

  var request:URLRequest=new URLRequest(remoteRequestURL);
  request.data=urlVariables;
  request.method=requestMethod;

  var urlLoader:URLLoader=new URLLoader();
  urlLoader.addEventListener(Event.COMPLETE, handleRequest);
  urlLoader.addEventListener(IOErrorEvent.IO_ERROR, handleRequestIOError);
  urlLoader.load(request);
}

We've looked here at some very useful EC2 API commands that can be used to programmatically start EC2 services on demand and to monitor the status of sets of EC2 instances. I've packaged up the complete example code which you can download here. You can use this as a starting point for developing your own Flex based AWS API library and using it in your own applications.


Splitting Django models and views into separate files.

A question I often hear when people work with the Django webapp framework is : How can I have one model per file and one view per file?

This is particularly understandable for people who have worked a lot with e.g. Ruby on Rails before coming to Django, like I did!

The thing is, it seems a lot cleaner and more natural to separate models and views into different modules rather than having them all stuffed into a couple of large files! However, unlike Rails, when you begin a Django webapp you get just two files, one for the models and one for the views. Let's suppose we have a Django app for gathering website statistics :

django-admin.py startapp webstats
  • webstats
    • __init__.py
    • models.py
    • tests.py
    • views.py

First thing is to delete the models.py and views.py files and create two directories : models and views

  • webstats
    • __init__.py
    • models
    • tests.py
    • views

Now let's suppose we have two models a Website and a Visitor, where Visitor has a foreign key relationship with Website.

Here's the code for our example Website class :

          
from django.db import models 

class Website(models.Model):
  domain = models.URLField("URL", max_length=200)

  class Meta: 
    app_label = 'webstats'

... and the visitor ...

          
from django.db import models 
from webstats.models.website import Website

class Visitor(models.Model):
  remote_addr = models.IPAddressField(blank=True)
  x_ff        = models.CharField(blank=True, max_length=100)
  time        = models.DateTimeField(auto_now_add=True, 
                                     editable=False)
  referer     = models.URLField(blank=True, verify_exists=False)
  user_agent  = models.CharField(blank=True, max_length=100)
  path        = models.CharField(blank=True, max_length=100)
  website     = models.ForeignKey(Website)

  class Meta:
    app_label = 'webstats'

The important thing to notice is the inclusion into to each model class of the inner meta class : 

          
class Meta:
  app_label = 'webstats'

The app_label is necessary so that the separated class definitions are correctly associated by syncdb with our webstats app.

Now we can import each model class into __init__.py :

  • __init__.py 
          
from webstats.models.website import Website
from webstats.models.visitor import Visitor

All that remains for the models now is to run syncdb :

          
python manage.py syncdb

That's it for the models! We now have a nice clean organistion of the model classes into their own separate files.

But we're not done yet! Let's turn now to the views ...

Let's suppose we have three view methods : index, webstats_main and track

Where index returns a list of monitored websites, webstats_main returns the collected stats for each website and track is a special view used for collecting the statistics ...

Now we just place each method into it's own file and finally, import the views in the __init__.py file :

We have :

  • views
    • __init__.py
    • index.py
    • webstats_main.py
    • track.py

... and in the __init__.py ...

          
from index import *
from webstats_main import *
from track import *

... and that's it! A separation of your model classes and views in Django. Have fun!


An n-body simulator written in JavaScript with html5 canvas

The canvas shows a dynamical system simulation using the fourth order Runge Kutta integration method. I wrote it using the html5 canvas element and JavaScript. I wanted to have a look at the performance when using RK4 in JavaScript and partly because I like playing with this sort of stuff!

The planet sizes reflect the relative masses and collisions with the boundary are inelastic subject to a damping factor.

All in all, the canvas looks very promising as an alternative to flash.

You can download a demo version of the source here the github repository