Quarkonium
IT Security

An MI5 Digital Intelligence puzzle

Hi! I came across this little puzzle recently that is an interesting exercise in pulling binaries out of tcp packet capture files. If you want to have a go on your own I won't spoil your fun, read no further and go to the MI5 web site.

The puzzle involves a tcp packet capture file containing data which can be extracted to reconstruct a rar archive file. The file is password protected and the password was also transmitted in the tcp session.

First get the file itjobs.pcap from the above site and install wireshark if you don't already have it.

The pcap file shows two smtp sessions, one where a password is sent in plain text and a subsequent session showing where an rar archive file was sent.

So let's look first at this password... open the pcap in wireshark and go to entry number 254, this is the start of the first smtp session tcp stream.

Select this entry and then in the 'Analyze' menu, click on 'Follow TCP Stream'. The result will be a window showing the entire conversation and as shown below, we only need the first part i.e.

Yes, we'll be putting that password to use later! Read on for the fun part... finding and reconstructing the rar archive!

The smtp session in question starts at entry number 491 ...

Again, under 'Analyze' click 'Follow TCP Stream'. We're interested in the first part of the conversation...

... and here's the data we're looking for, a base64 encoded stream that we'll now extract and decode.

With the 'RAW' option selected, click 'Save As' and open the resulting file in your favourite editor. We only want the data stream so delete all of the header info just leaving the data block and nothing else so it will look like this

...

Here I've removed all lines before and after the base 64 data block, note that explicit carriage returns were also removed!

We can now decode our base 64 data, a bit of python will suffice!

          
import base64

#data_base64 our file containing the base64 data
input_file = open("data_base64", "r")
s = input_file.readline()
input_file.close()

#remove the superfluous carriage return 
#added by the readline function!
s = s.strip()
s = base64.decodestring(s)

#our reconstructed rar archive!
output_file = open("backup.rar", "wb")
output_file.write(s)
output_file.close()
          
        

With that we have our rar archive backup.rar

Now we can see what's inside, the file is password protected and you've guessed it this happens to be the one we found earlier!

So extracting using the password Hell0W0rld!

and that's it, we've successfully reconstructed the rar archive and extracted the four files contained therein! Your probably wondering what's in them, well two of them contain some 'Well done ... only three of the files are hiding something ...' text and one (remember.docx) is empty. The file 'info.docx' contains a supposed list of files of interest on the user's disk

Have fun!

Form posts using PHP with cURL and Tor
... or posting via anonymous proxy ...

Hi! I'll give an example here of using PHP curl to test website forms over anonymous proxy using Tor.

Tor (originally short for The Onion Router) is a worldwide network of relays used to avoid surveillance and traffic analysis.

Let's assume you have Tor installed and running on the default port 9050. In the example code that follows we first setup the curl options then post some parameters to our form using curl_exec. After examining the response we decide whether to continue with more iterations or not...

          
<?php
  //Contains our test data array ...
  require_once("test_array.php");

  //URL   
  $url = $FORM_URL;

  //Headers
  $headers = array(
    'Host: $FORM_HOST',
    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Referer: $FORM_URL'
  );
    
  //Tor socket address, default port is 9050
  $tor = '127.0.0.1:9050';

  //curl
  $ch = curl_init();
   
  //Set the proxy address
  curl_setopt($ch, CURLOPT_PROXY, $tor);
   
  //Set the proxy type
  curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);

  //The URL of the form we're posting to 
  curl_setopt($ch, CURLOPT_URL, $url);
   
  //Set request headers
  curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
   
  //Prepare for the POST operation
  curl_setopt($ch, CURLOPT_POST, 1);
   
  //Follow any "Location: " header that the server sends
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
   
  //Don't return HTTP headers
  curl_setopt($ch, CURLOPT_HEADER, 0);
   
  //Return the contentof the call
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
  
  //Loop over an array of test data ... 
  foreach($array as $key => $name)
  {
    //Set POST fields, for example ...
    $data = array (
      "email" => $SOME_CONTACT_EMAIL,
      "Address" => $REGISTRATION_ADDRESS,
      "Country" => $COUNTRY,
    );
 
    //traverse our test array (from require_once) and 
    //prepare the data for posting (key1=value1)
    foreach( $data as $key => $value) 
    {
      $post_items[] = $key . '=' . $value;
    }

    //Using implode(), create the post parameter string ...
    $post_string = implode ('&', $post_items);
   
    $post_items = array();

    print_r("$post_string\n");

    curl_setopt($ch, CURLOPT_POSTFIELDS, $post_string);

    $result = curl_exec($ch);

    echo "result $result\n";

    //E.g. test the response for the presence of some field...
    if (strpos($result, $PATTERN) === false) 
    {
      print_r("Found expected pattern $PATTERN in response");
     
      die('OK');
    }

    //Continue test ...
    print_r("Tested fields, continue \n");
  } 
?>
          
        

Have fun!

Using the AWS EC2 API in Flex

Welcome! This month I'm going to give a quick tutorial elucidating some basic and very useful AWS EC2 API calls from within Flex. At the end, you can download some example code which you can use as a starting point in building your own AWS API library for Flex! For this tutorial you'll need an AWS account, if you haven't already got one then you can sign up here you can then create and configure an EC2 micro instance using the free usage tier.

In the name of Iron Maiden! Let's begin!

Amazon Web Services provides two types of API for controlling our EC2 instances one is SOAP based the other is REST based, we'll be using the latter in this tutorial

Let's assume we have our EC2 instance set up. Now in your AWS account navigate to your EC2 dashboard. You should see your list of configured EC2 instances. In my test I'm using the server below named 'ec2-server'

Notice the Instance column above, this field contains the instance id that is used to identify our EC2 instance. We'll be using this in the code that follows.

Let's look at how we can programmatically examine the status of any our instances. For this we're going to use the DescribeInstances API call which takes an optional instance id. Supplying the instance id returns information on just that particular instance, while supplying no instance id returns the status of *all* of our instances.

Below is the xml returned in my test set up as a result of the DescribeInstances call ...

    
<DescribeInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2012-12-01/">
  <requestId>2050b11e-2a9a-419a-b8ae-73249f0a7878</requestId>
  <reservationSet>
    <item>
      <reservationId>r-52ef3136</reservationId>
      <ownerId>393619079936</ownerId>
      <groupSet>
        <item>
          <groupId>sg-b08914d8</groupId>
          <groupName>quick-start-1</groupName>
        </item>
      </groupSet>
      <instancesSet>
        <item>
          <instanceId>i-eaedd492</instanceId>
          <imageId>ami-aecd60c7</imageId>
          <instanceState>
            <code>80</code>
            <name>stopped</name>
          </instanceState>
          <privateDnsName/>
          <dnsName/>
          <reason>User initiated (2013-02-01 18:31:16 GMT)</reason>
          <keyName>test_server</keyName>
          <amiLaunchIndex>0</amiLaunchIndex>
          <productCodes/>
          <instanceType>t1.micro</instanceType>
          <launchTime>2013-02-01T18:24:26.000Z</launchTime>
          <placement>
            <availabilityZone>us-east-1c</availabilityZone>
            <groupName/>
            <tenancy>default</tenancy>
          </placement>
          <kernelId>aki-88aa75e1</kernelId>
          <monitoring>
            <state>disabled</state>
          </monitoring>
          <groupSet>
            <item>
              <groupId>sg-b08914d8</groupId>
              <groupName>quick-start-1</groupName>
            </item>
          </groupSet>
          <stateReason>
            <code>Client.UserInitiatedShutdown</code>
            <message>Client.UserInitiatedShutdown: User initiated shutdown</message>
          </stateReason>
          <architecture>x86_64</architecture>
          <rootDeviceType>ebs</rootDeviceType>
          <rootDeviceName>/dev/sda1</rootDeviceName>
          <blockDeviceMapping>
            <item>
              <deviceName>/dev/sda1</deviceName>
              <ebs>
                <volumeId>vol-08cee669</volumeId>
                <status>attached</status>
                <attachTime>2012-07-25T13:50:11.000Z</attachTime>
                <deleteOnTermination>true</deleteOnTermination>
              </ebs>
            </item>
          </blockDeviceMapping>
          <virtualizationType>paravirtual</virtualizationType>
          <clientToken>ozLCT1343224207944</clientToken>
          <tagSet>
            <item>
              <key>Name</key>
              <value>ec2-server</value>
            </item>
          </tagSet>
          <hypervisor>xen</hypervisor>
          <networkInterfaceSet/>
          <ebsOptimized>false</ebsOptimized>
        </item>
      </instancesSet>
    </item>
  </reservationSet>
</DescribeInstancesResponse>
    
  

Notice the instanceState block wtihin the instanceSet showing the instance status.

    
          <instanceState>
            <code>80</code>
            <name>stopped</name>
          </instanceState>
    
  

Let's look at the code, our demo library contains an EC2 class which handles the execution of the various API calls using the RESTful interface. In the calling class EC2APIDemo below we instantiate the EC2 class and set up the handler to process the results.

    
public function EC2APIDemo()    
{
  var AWS_ACCESS_ID:String = "YOUR_AWS_ACCESS_KEY";      
  var AWS_SECRET_KEY:String = "YOUR_AWS_SECRET_ACCESS_KEY";      
  var AWS_EC2_SERVER_INSTANCE_ID:String = "i-eaedd492";
 
  var ec2Interface:EC2 = new EC2(EC2.US_EAST_1);
  ec2Interface.setAWSCredentials(AWS_ACCESS_ID,
                                AWS_SECRET_KEY);
        
  var parameterArray:Array = new Array();
  parameterArray.push(new Parameter("InstanceId.1", 
                                     AWS_EC2_SERVER_INSTANCE_ID));
        
  ec2Interface.addEventListener(AWSEvent.RESULT, handler);      
  ec2Interface.executeRequest(EC2.DESCRIBE_INSTANCES, parameterArray);
}
    
  

Here's the handler, all we're doing for now is iterating over the ids and the associated state

    
public function handler(event:AWSEvent):void
{
  var xmlData:XML = new XML(event.data);
  trace("handler " + xmlData);
  for each(var item:* in xmlData.*::reservationSet.*::item.*::instancesSet.*::item)
  {
    trace("  item.*::instanceState.*::name " + item.*::instanceState.*::name);
    trace("  item.*::instanceId " + item.*::instanceId);
  }
}
    
  

This brings us to our next EC2 API call: Having examined the instance state and finding that it's stopped, we can make use of the StartInstances API calL to get the instance running. As you can see looking at the docs, this expects one or more instance ids so you can start up a whole cluster of them if you want!

    
public function startInstances():void
{
  var parameterArray:Array = new Array();
  parameterArray.push(new Parameter("InstanceId.1", AWS_EC2_SERVER_INSTANCE_ID));

  this.ec2Interface.addEventListener(AWSEvent.RESULT, handlerStartInstances);
  this.ec2Interface.executeRequest(EC2.START_INSTANCES, parameterArray);
}
    
  
    
public function handlerStartInstances(event:AWSEvent):void
{
  var xmlData:XML = new XML(event.data);
  trace("handlerStartInstances " + xmlData);
 
  //Do stuff ... 
  //Error handling ...
}
    
  

The response from the StartInstances call as described in the API, results in a list of instance state changes. In this case the state is changed to 'pending' i.e. startup is in progress ...

    
<StartInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2012-12-01/">
  <requestId>ef28014b-4fbc-408f-85b9-3d09b9e655d4</requestId>
  <instancesSet>
    <item>
      <instanceId>i-eaedd492</instanceId>
      <currentState>
        <code>0</code>
        <name>pending</name>
      </currentState>
      <previousState>
        <code>80</code>
        <name>stopped</name>
      </previousState>
    </item>
  </instancesSet>
</StartInstancesResponse>
    
  

Taking a look under the hood in the demo Flex EC2 API library we have the method executeRequest. Here we can see the url request constructed using the given EC2 API action e.g. DescribeInstances, StartInstances, StopInstances ...

    
public function executeRequest(action:String, urlVariablesArr:Array=null, requestMethod:String="POST"):void
{
  this.action = action;
  if(!urlVariablesArr)
  {
    urlVariablesArr = new Array();
  }

  urlVariablesArr.push(new Parameter("Version", "2012-12-01"));
  urlVariablesArr.push(new Parameter("Action", action));
  urlVariablesArr.push(new Parameter("AWSAccessKeyId", awsAccessKey_));
  urlVariablesArr.push(new Parameter("Timestamp", 
                                     AWSDateUtil.generateRequestTimeStamp(new Date())));

  var urlVariables:URLVariables=generateSignature(urlVariablesArr, 
                                                  signatureVersionToUse, 
                                                  requestMethod);
  for each (var item:Parameter in urlVariablesArr)
  {
    urlVariables[item.key]=item.value;
  }

  var request:URLRequest=new URLRequest(remoteRequestURL);
  request.data=urlVariables;
  request.method=requestMethod;

  var urlLoader:URLLoader=new URLLoader();
  urlLoader.addEventListener(Event.COMPLETE, handleRequest);
  urlLoader.addEventListener(IOErrorEvent.IO_ERROR, handleRequestIOError);
  urlLoader.load(request);
}
    
  

We've looked here at some very useful EC2 API commands that can be used to programmatically start EC2 services on demand and to monitor the status of sets of EC2 instances. I've packaged up the complete example code which you can download here. You can use this as a starting point for developing your own Flex based AWS API library and using it in your own applications.

Splitting Django models and views into separate files.

A question I often hear when people work with the Django webapp framework is : How can I have one model per file and one view per file?

This is particularly understandable for people who have worked a lot with e.g. Ruby on Rails before coming to Django, like I did!

The thing is, it seems a lot cleaner and more natural to separate models and views into different modules rather than having them all stuffed into a couple of large files! However, unlike Rails, when you begin a Django webapp you get just two files, one for the models and one for the views. Let's suppose we have a Django app for gathering website statistics :

django-admin.py startapp webstats
  • webstats
    • __init__.py
    • models.py
    • tests.py
    • views.py

First thing is to delete the models.py and views.py files and create two directories : models and views

  • webstats
    • __init__.py
    • models
    • tests.py
    • views

Now let's suppose we have two models a Website and a Visitor, where Visitor has a foreign key relationship with Website.

Here's the code for our example Website class :

          
from django.db import models 

class Website(models.Model):
  domain = models.URLField("URL", max_length=200)

  class Meta: 
    app_label = 'webstats'
          
        

... and the visitor ...

          
from django.db import models 
from webstats.models.website import Website

class Visitor(models.Model):
  remote_addr = models.IPAddressField(blank=True)
  x_ff        = models.CharField(blank=True, max_length=100)
  time        = models.DateTimeField(auto_now_add=True, 
                                     editable=False)
  referer     = models.URLField(blank=True, verify_exists=False)
  user_agent  = models.CharField(blank=True, max_length=100)
  path        = models.CharField(blank=True, max_length=100)
  website     = models.ForeignKey(Website)

  class Meta:
    app_label = 'webstats'
          
  

The important thing to notice is the inclusion into to each model class of the inner meta class :

          
class Meta:
  app_label = 'webstats'
          
  

The app_label is necessary so that the separated class definitions are correctly associated by syncdb with our webstats app.

Now we can import each model class into __init__.py :

  • __init__.py
          
from webstats.models.website import Website
from webstats.models.visitor import Visitor 
          
  

All that remains for the models now is to run syncdb :

          
python manage.py syncdb 
          
  

That's it for the models! We now have a nice clean organistion of the model classes into their own separate files.

But we're not done yet! Let's turn now to the views ...

Let's suppose we have three view methods : index, webstats_main and track

Where index returns a list of monitored websites, webstats_main returns the collected stats for each website and track is a special view used for collecting the statistics ...

Now we just place each method into it's own file and finally, import the views in the __init__.py file :

We have :

  • views
    • __init__.py
    • index.py
    • webstats_main.py
    • track.py

... and in the __init__.py ...

          
from index import *
from webstats_main import *
from track import *
          
  

... and that's it! A separation of your model classes and views in Django. Have fun!

An n-body simulator written in JavaScript with html5 canvas

The canvas shows a dynamical system simulation using the fourth order Runge Kutta integration method. I wrote it using the html5 canvas element and JavaScript. I wanted to have a look at the performance when using RK4 in JavaScript and partly because I like playing with this sort of stuff!

The planet sizes reflect the relative masses and collisions with the boundary are inelastic subject to a damping factor.

All in all, the canvas looks very promising as an alternative to flash.

You can download a demo version of the source here the github repository