My IP geo location script cache requests coming from the same ip range (subnet mask 255.255.255.0) named as class-c ip address (last number cutted). This value is the reference for look up in memcache. But yesterday I got an exception while calculating the ip from the request object.

def get(self):
    ipStr = self.request.remote_addr
    classC = reduce(lambda a, b: int(a)*256+int(b), ipStr.split(".")[:3])
    logging.info("request from classC ip address:: %d" % classC)
    ...

This request crashed (TypeError: int argument required) by using an IPv6 address (found in the google app engine logs):

03-18 05:00PM 37.126 /geo_data.js 500 35ms 38cpu_ms 0kb Mozilla/5.0 (Windows; U; Windows NT 6.1; pt-PT; rv:1.9.2) Gecko/20100115 Firefox/3.6,gzip(gfe)
2a01:e35:2f20:f770:6c54:xxx:67fb:df8 – – [18/Mar/2010:17:00:37 -0700] “GET /geo_data.js HTTP/1.1” 500 450 “http://www.xxx.com.br/spl19/index.php?refid=cm_av_ri” “Mozilla/5.0 (Windows; U; Windows NT 6.1; pt-PT; rv:1.9.2) Gecko/20100115 Firefox/3.6,gzip(gfe)”

The typical log line looks like this:

03-19 02:12PM 23.047 /geo_data.js 200 411ms 272cpu_ms 175api_cpu_ms 0kb Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1036 Safari/532.5,gzip(gfe)
189.55.xxx.160 – – [19/Mar/2010:14:12:23 -0700] “GET /geo_data.js HTTP/1.1” 200 770 “http://www.xxx.com.br/spl17/index.php?refid=gv_av_ge” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1036 Safari/532.5,gzip(gfe)”

(A part of the complete ip address is changed)

Since March 8, 2010 App Engine joins the Google over IPv6 Program and the time is over to parse ip address with the typical ip address regex.

If you want to calculate a number from a IPv6 address try this short functional python code:

def ipToInt(ipStr):
    if ipStr.find(".")!=-1:
        return reduce(lambda a, b: int(a)*256+int(b), ipStr.split("."))
    else:
        return reduce(lambda a, b: a*65536+b, [int(x or "0",16) for x in ipStr.split(":")])

And I should check if the ip-to-geo-location provider offers his mapping with IPv6 adresses:

• try it with ipInfoDb: Invalid IP address or domain name.
• try it with geoPlugin.net: Invalid IP address
• try it with hostIp: Location: Private block address.

*ups* – no one supports the IPv6 !

Generating thumbnails from a collection of pictures is an easy task and there are many tools for the job. But I didn’t find a tool who can clip out a centered box of my pictures to get quadratic thumbs of my originals. Because I am a developer and I dont want to try out many tools: I will hack a short python script (with help of the PIL) myself.

simple solution

The following python script can be find on many webpages. It will take all command line arguments as image filenames and convert it to 200×200 thumbnails with the default extension “_thumb.jpg” (build from the original filename):

import Image, os, sys

for filename in sys.argv[1:]:
    img = Image.open(filename).resize( (200,200) )
    out = file(os.path.splitext(filename)[0]+"_thumb.jpg", "w")
    try:
        img.save(out, "JPEG")
    finally:
        out.close()

With the command line tool find all images and converted:

find . -regex .*jpg | xargs python resize.py

But the result is not ok:

• the only good part: the result image is 200×200 pixel
• aspect ratio ignored
• ugly quality
• performance bad: 1.025 sec (example image on my eeeBox)
• no value to choose the jpg quality for saving

For my web gallery I need identical thumbnails for a clean design and the aspect ratio should not be ignored!

Challenge: Find the biggest box with correct aspect ratio and clip the image, resize it fast and with the best quality to the given thumbnail size!

resize tips you should know

When choosing an image libary (with python is the build-in variant PIL good enough, but Samuel will add an article for java image libraries) for resizing images try to find out the following three parts:

choose the right algorythm

There should be a option to choose the image transforming algorithm. “Fast and ugly” or “slow and high quality”.

With the python imaging library there are included algorithm for image translations (ordered from fastest to best quality): NEAREST, BILINEAR, BICUBIC, ANTIALIAS. If you resize an image with NEAREST the quality will be poor but the difference in calculation time to ANTIALIAS is factor 3 (depending on images). The first example has the NEAREST as default value.

img = Image.open(filename).resize( (200,200), Image.ANTIALIAS)

Resizing the image with the best algorithm will produce a good quality, but needs more than 3 sec.

use two resize steps

The trick for resizing an 1600×1600 image to 200×200:

• first resize it from 1600×1600 to double the target size (400×400) with the fastest allgorithm
• resize the smaller image to the target resolution with the best algorithm

You get the fast transformation for the big amount and the last step to the target resolution with the best quality.

img = Image.open(filename).resize( (400,400) )
img = img.resize( (200,200), Image.ANTIALIAS)

This 2-pass resize part runs in exactly the same time (1.02 sec) like the first example with better quality.

resize vs. thumbnail

Because our picture collection contain 90% JPEG there is an other optimization step. To resize JPEGs you dont need all pixels for downsizing with factor 8. Its adequate to read only the starting point of an minimum coded Unit (MCU -> wikipedia). You have to try out if it is supported of your image lib – the transformation time is the answer.

The python image library offer this option with the Image.thumbnail-method. An other aspect is that the thumbnail parameter mean a target box size instead the target image size – so the aspect ratio will be saved.

img = Image.open(filename)
img.thumbnail( (200,200) )

The thumbnail method works on the image object and dont produce a new image object. Its faster: 0.25 sec.

img = Image.open(filename)
img.thumbnail( (400,400) )
img.thumbnail( (200,200), Image.ANTIALIAS)

The two-pass variant is slower but has better quality on the edges. Its runs 0.42 sec.

other problems

You should be careful by loading every image. I can produce a pixel bomb by saving an 40000×40000 pixel black picture as a png. This image file is not expensive but your memory will not hold the complete bitmap. Check the resolution or have an idea to handle the OutOfMemory-exception – before loading!

There are many corrupt example images. Try these images to check if your image library is robust.

benchmark for the different steps

• 1.02 sec: direct resize with bad quality
• over 3 sec: direct resize with best quality
• 1.02 sec: two resize steps with best quality
• 0.25 sec: direct thumbnail call with bad quality
• 0.42 sec: two thumbnail calls with best quality

This is factor 8 for different, best quality resize operation and worth for code-checking. If you plan to resize images on-the-fly for webpages there are other technics than these basic tips …

Finding the center box

I will construct two extreme examples for defining the clipping algorithm. If the two examples works the quadratic thumbnail work too.

Get from a portait picture a landscape thumb

• Original: bmw.jpg (2048×3072 pixel)
• thumbnail-box: 200×100 pixel

The clipping algorithm should find the centered box:

This picture is ideal for a wide thumbnail because the dominant radiator grill of the old bmw is THE part of the picture!

Get from a landscape picture a portait thumb

• Second Example: ford.jpg (3072×2048 pixel)
• thumbnail-box: 100×200 pixel

The clipping algorithm should find the centered box:

This old ford T-Model image has to many people on the image – a portrail thumbnail crop some from the front.

the complete function

def resize(img, box, fit, out):
    '''Downsample the image.
    @param img: Image -  an Image-object
    @param box: tuple(x, y) - the bounding box of the result image
    @param fix: boolean - crop the image to fill the box
    @param out: file-like-object - save the image into the output stream
    '''
    #preresize image with factor 2, 4, 8 and fast algorithm
    factor = 1
    while img.size[0]/factor > 2*box[0] and img.size[1]*2/factor > 2*box[1]:
        factor *=2
    if factor > 1:
        img.thumbnail((img.size[0]/factor, img.size[1]/factor), Image.NEAREST)

    #calculate the cropping box and get the cropped part
    if fit:
        x1 = y1 = 0
        x2, y2 = img.size
        wRatio = 1.0 * x2/box[0]
        hRatio = 1.0 * y2/box[1]
        if hRatio > wRatio:
            y1 = int(y2/2-box[1]*wRatio/2)
            y2 = int(y2/2+box[1]*wRatio/2)
        else:
            x1 = int(x2/2-box[0]*hRatio/2)
            x2 = int(x2/2+box[0]*hRatio/2)
        img = img.crop((x1,y1,x2,y2))

    #Resize the image with best quality algorithm ANTI-ALIAS
    img.thumbnail(box, Image.ANTIALIAS)

    #save it into a file-like object
    img.save(out, "JPEG", quality=75)
#resize

Additional tip: Cropping from the smaller image will save memory and memcopy-operation.

• Line 13: fast pre-resize
• Line 27: cropping the pre-sized image with the calculcated cropping values
• Line 30: resize the image to the target box (aspecti ratio with using of method thumbnail is considerd).
• Line 33: Saving a thumbnail with 75% is acceptable for the quality

Download area

• the complete script with doctest and command line option is includet here in the article for free download.
• the bmw and ford t model pictures are taken while the Leo Classic 2009 (Leonberg – Germany) and are for free use (download here).

How to do this in java

The questions for efficent and high quality resizing with java libs is open for the next article … be free to give improvements in the comments or trackbacks!

This article will describe how to build a more complex javascript application. I chose to build a number cruncher for finding sociable numbers. There are many idle browser out there and the new Worker-thread object (available in modern browsers) offers the complete power of all cpu cores. In the first part I will describe how to inherit javascript objects with the optimization steps and discuss the usage of a profiler. The second part will improve the performance with using Web Worker in threads for greater computing power and how to handle it.

introduction

Did you heard of perfect numbers? No? A number is perfect if the number is equal to the sum all proper positive divisors. The old greek mathematics found the following examples (or try out the demo):

The 6 has as positive divisors 1, 2 and 3 and the sum is 6.
The next perfect number is 28 with 1, 2, 4, 7 and 14
The 496 = 1 + 2 + 4 + 8 + 16 + 31 + 62 + 124 + 248
The 8128 = 1 + 2 + 4 + 8 + 16 + 32 + 64 + 127 + 254 + 508 + 1016 + 2032 + 4064.

Funny but not easy to find more … Amicable numbers were known to the Pythagoreans and are two different numbers so related that the sum of the proper divisors of one of the numbers is equal to the other. For example, the smallest pair of amicable numbers is (220, 284); for the proper divisors of 220 are 1, 2, 4, 5, 10, 11, 20, 22, 44, 55 and 110, of which the sum is 284; and the proper divisors of 284 are 1, 2, 4, 71, and 142, of which the sum is 220.

If the cycle to reach the starting number (with the recursion of the sum of positive divisors) and the length of the cycles is greater than 2 they called sociable or multiperfect numbers”. Its a more general concept for the first two special cases.

If you are interested in the mathemical background there are many resources in the net:

• The Perfect Number Journey
• a little history about perfect numbers
• great source of Multiply Perfect Numbers from Achim Flammenkamp
• the book Unsolved Problems in Number Theory by Richard Guy
• Algorithms in the Study of Multiperfect and Odd Perfect Numbers by Sorli, Ronald

the motivation

Because no general formula exists to find all sociable numbers it would be nice to find a new cycle at first. There is some formula to find perfect numbers based on prime numbers but I will build only a number cruncher with testing a given range for perfect numbers. There are many other distributed computer projects (like the Great Internet Mersenne Prime Search) why not this part of number theory? But don’t think this is an easy task. After the first 200 years all mathematicians found only a few cycles. With the modern computing power all cycles under 100.000.000 were found – see the complete list at http://djm.cc/sociable.txt.

And I want to solve this in your browser – with javascript.

implementation

the algorithm

The com.unitedCoders.Numbers1 object realized the algorithm straight forward:

var com = com||{};
com.unitedCoders = com.unitedCoders||{};

com.unitedCoders.Numbers1 = function(){
    this.maxLoop = 100;
    /* calculate all divisors of n and return the sum of it
    @var n int  - number to be checked
    @return the sum of all divisors
    */
    this.calcSumOfDivisors = function(n) {
      //include the divisor 1
      var sum = 1;
      var middle = Math.sqrt(n);
      for (var i=2; i <= middle; i++) {
        if (n % i === 0) {
        sum += i;
        sum += n/i;
        }
      }
      return sum;
    };

The calcSumOfDivisors function finds the integer divisors with the modulo operator. The loop runs only up the square of the number because you can get directly the other of the divisor pair. For example if you found 3 is a divisor of 30 so is 30/3=10 a divisor too.

    /* generate a cycle and break if maxLoop reached or n=1 or n is too great
    @var start int - start number of loop
    @return [int] - calculated loop
    */
    this.calcCycle = function(start) {
      var last = start, f = [], max = 100*start, length;
        do {
            last = this.calcSumOfDivisors(last);
        length = f.push(last);
      } while (start!==last && last>1 && length<this.maxLoop && last<max);
      return f;
    };

To find a cylce of the sum of all divisors you can start recursively until getting the starting number. I chose a simple while-do loop ending with the starting number or break after 100 tries. If the found cycle ends with length of 1 it is a perfect number, with 2 a pair of amicable numbers are found and if the cycles end with more the 2 numbers the result could be sociable numbers. The largest known cycle at this time is 28. To limit the loop I will be pessimistic and try up 100 runs until my function gives up.

    /* Calculate the cycle, check if there is a loop and if n the first element,
       return the numbers as a list.
    @var n int - start number for checking
    @return [int] if n is starting point of a social number group
    */
    this.getMinSociable = function(n) {
        var f = this.calcCycle(n, this.maxLoop);
        // Check if it has a cycle
    if (n === f[f.length-1]) {
        //Check if n is the lowest member of the cycle
        var sorted = f.slice();
        sorted.sort(function(a,b){return a-b;});
            if (n === sorted[0]) {
                //correct the list, put the starting number at pos:0
                f.unshift(f.pop());
                return f;
            }
        }
        return;
    };
    /* Find sociable numbers from start to end
    @var start int - Starting number for checking
    @var end int - ending point for checking
    @return [ [int], ] list of sociable groups
    */
    this.findSociable = function(start, end) {
        var results = [];
        for (var i=start; i<end; i++) {
            var match = this.getMinSociable(i);
            if (match) {
                results.push(match);
            }
        }
        return results;
    };
};

The function getMinSociable() check if the result of calcCycle are sociable numbers and if is starts with the lowest number to get only one result. The seconds condition will prevent to find a cycle more the once.

first optimization step

If you running the functions in firefox with firebug you can profile all the function calls (since firebug 1.4.x the function count could be incorrect but the tendency is ok). Open the firebug console and start a search between 1 and 15.000. I got these function call list with executing times.

This is the result in firefox 3.5 in my tiny eeeBox with the typical netbook processor (Intel Atom n270 mobile processor with 1.6 MHz - to compare the times).

firefox 3.5, profiler output with my intel atom cpu (wrong counting of the function calls)

firefox 3.5, profiler output with my office intel p4 cpu

The run for 15.000 cycles takes 5.4 sec (3,4 sec on my P4 cpu) - most time in the deepest function calcSumOfDivisors. You notice that for checking 15000 numbers the calcSumOfDivisors-function will be called over 175.000 times. Many cycles end will 1 and a simple caching for the calculation would decrease the calls and speed up.

Caching before calculation

If a cache call is faster than a calculation it could be a good way for better performance. If the cache hit rate can save some calculation time do caching! In the example would a simple JavaScript object help to decrease the calcSumOfDivisors-function. The calcCycle-function will call the caching facade.

com.unitedCoders.numbers.getSumOfDivisors : function(n) {
    var sum = this.sumCache[n];
    if (sum) { return sum; }
    
    sum = this.calcSumOfDivisors(n);
    this.sumCache[n] = sum;
    return sum;
};

firefox 3.5, profiler output with my intel atom cpu

firefox 3.5, profiler output with my office intel p4 cpu

The overall time is only 4 sec (0.9 sec on my P4 cpu) and the number of calcCumOfDivisor decreased below to 20.000! I added a direct cache look-up in the calcCycle function to decrease the function calls so the getSumOfDivisor is only for "cache miss and saving the result" used.

no function calls

The result of the third step looks ugly. Because JavaScript runs in a interpreter eliminating all function calls could increase the calculator speed, but your code will become unreadable. This is only a try to test if it works.

This should save the time to calling the functions and initializing the local varibale scope. An other aspect is giving the JavaScript runtime optimizer a large block to work with it.

com.unitedCoders.Numbers3 = function(){
    this.findSociable = function(start, end) {
        var maxLoop = 100;
        var results = [];
        for (var n=start; n<end; n++) {
            //calcCycle
            var lastN = n, f = [], maxN = 100*n, length;
            do {
                //getSumOfDivisors
                if (!this.sumCache[lastN]) {
                    this.sumCache[lastN] = this.calcSumOfDivisors(lastN);
                }
                lastN = this.sumCache[lastN];
                length = f.push(lastN);
            } while (n!==lastN && lastN>1 && length<this.maxLoop && lastN<maxN);

            //getMinSociable
            if (n===lastN) {
                var sorted = f.slice();
                sorted.sort(function(a,b){return a-b;})
                if (n === sorted[0]){
                    f.unshift(f.pop());
                    results.push(f);
                }
            }
        }
        return results;
    };
};
com.unitedCoders.Numbers3.prototype = new com.unitedCoders.Numbers2();

firefox 3.5, profiler output with my intel atom cpu

firefox 3.5, profiler output with my office intel p4 cpu

Result: Wow - the firefox 3.5 runtime optimizer can work well with the one function version! The older firefox 3.0 show no benefit in this version - see the benchmark in the next section.

Use Webworker

Webworker is a new HTML5 feature supported by modern browsers since mozilla firefox 3.1, apple webkit or google chrome. It does not work on opera, firefox 3.0 or konqueror. The worker is a JavaScript object in a extra thread who can communicate with the main application via the function postMessage.

For the simple example I created a a "worker hook function" in my numbers.js file and started the cruncher object synchronous like in the normal version. The final version can split the number range in parts for each worker thread.

   worker = new Worker("numbers.js");
   worker.onmessage = function(resultStr) { window.alert(resultStr); }
   worker.postMessage("3,1,15000");

The Worker object is loaded, will start in background the calculation and give the result as json-string to the callback function (which shows as ugly alert box).

//this will load the local json2-file from crockford, got from http://json.org/
importScripts("json2.js");
//this is the called function from worker.postMessage
//@param {String} "variant,from,to"
onmessage = function(params) {
    var p = params.data.split(",");
    var worker = new com.unitedCoders['Numbers'+p[0]]();
    var results = worker.findSociable(+p[1], +p[2]);
    //this is the calling function for worker.onmessage
    postMessage(JSON.stringify(results));
};

I expect better results with using more than one worker. Best would be one worker per cpu core. Chrome scales perfectly because the browser has an extra worker process for every tab! Firefox has one process but I can see more the one working thread in top.

Get modern browser running in ubuntu

If you using ubuntu (or other apt-based linux) and don't have the newest firefox (or didn't upgrad to ubuntu 10.x), here is the command to get it directly. Firefox can be installed via

 > sudo apt-get install firefox-3.5

.

Because googles browser chrome is calling home, use the privacy cleaned version Iron chrome from SRWare (see Review here and here).

Or get the normal version via apt-get following the steps from ubuntugeek.com.

The other two big player apple safari and internet explorer won't run natively under ubuntu. I will try to install safari with wine hoping the core JavaScript speed doesn't differ. And for the internet explorer there is the fine ie4linux project.

Benchmarking the variants

How to benchmark the code? I created the three evolutionary code versions and three work modes. First the directly variant as normal JavaScript function, second run the same function in a Webworker in a thread. The third version will split the calculation in 4 parts and start 4 webworker to hit the score.

For testing the variants in different browser I choosed on my ubuntu eeeBox the following browser, because all candidates use different javascript engines:

results

My optimization in three variants works on every browser and result caching is OK. V8 will optimize my single function variant dramatically - the run-time JavaScript optimizer works great! The starting and communication overhead for the worker is reasonable and it can benefits while computing more the one second per thread.

The four thread worker version was not faster than the normal version? OK, with only one core on one cpu (my intel Atom) could threaded number crunching not scale! Testing on a four core CPU: Yes it works fine but not linear, try it out in the interactive demo (open the firebug console for logging info)!

But don't forget your old-stuffed visitors (or internet explorer users): If your JavaScript calculations runs in chrome fine - your visitors with older browser will be frustrated! Factor 10 in calculation time is possible.

After the fine and long article about escaping from Matthias here some examples for special characters in a simple web application. This article should be only an inspiration, I will describe some code samples with python/javascript and explain why [

<"@%'&_\?/:;,>?? €

] is the ultimate input to test input in web applications.

The demo application offers a simple form with name and message field for an one-line guest book.

The examples will start with three lines. First the input data (which can testet in the demo app), second the escaped values in the transport data and a hint, why the escaping is needed.

Escaping in url parameters

To send values via urls use a simple formular and your browser will do all for you. For testing use these special values are &, ?, =, # and the space.

1. Input: [Barnes&Noble] says [TOP #10 question?]
2. used transport url is:

   ... resource?who=Barnes+%26+Noble&what=TOP+%2310+question%3F
   

3. If you forget to escaping the “#” the url parameter ended and the url fragment part starts.

parameters in a url

A simple html formular for sending with the GET-method:

<form id="form" method="GET">
  Name: <input id="who" name="who" size="10"> says <input name="what"/>
  <input type="submit" value="Send as formular with GET-parameter"/>
</form>

Because the browser sends the values your dont need to do something special.

Unescaping the url parameter in python

Dont try to parse the parameter directly from the ready build url. In the python webapp.RequestHandler has the right helper function:

class MessageExampleApp(webapp.RequestHandler):
    def get(self):
        data = [self.request.get("who", ""), self.request.get("what", "") ]
    #get
#MessageExampleApp

special characters in JSON

If you use for sending (ajax-request) the data in the json format the special characters will be control characters and the “.

1. Input:[Barnes & Noble] says [Peter cry “Yahoo”]
2. the generated json code is:

   {"who":"Barnes & Noble","what":"Peter cry \"Yahoo\""}
   

3. If you forget to escape the ” the json could be incorrect and can not parsed.

send json with javascript & jQuery

To generate the complete data as a json string, do not simply adding the values. I use the json-lib from json.org, because the actual jQuery lib dont offers the stringify-function. To send the data simply use the $.ajax() function.

<script type="text/javascript" src="//pyunitedcoders.appspot.com/json2.js"></script>
<script type="text/javascript">
  var data = {who: $('[name=who]')[0].value, what:$('[name=what]')[0].value};
  $.ajax({ type: "POST",
           data : JSON.stringify(data),
           contentType : "application/json; charset=utf-8",
           url: "...", dataType:"json", success: insertData });
})
</script>

insert json data into the document

Getting data in json format from a web service is a common task (and jQuery will callback the function insertData with ready parsed data). But if you insert the data in the document there is the next escaping pitfall. I used jQuery for the response and node.text() instead node.html() for inserting the data – the text()-function will escape problematic characters.

function insertData(res) {
    $("#msg").text(res.who + ' says "' + res.what + '"');
};

Escaping in html

After getting data from a database and inserting in html, you should take care of html injection (special characters are <, > and &)!

1. Input: [Barnes & Noble] says [<TOP10>]
2. after including in the result page the html code is

   Barnes & Noble says "<TOP 10>"
   

3. If you forget to escape the characters < and > the message will become an html tag and (mostly) invisible!

escape the data in a django template

How to solve the problem? I used the escape filter in the basic django template.

  <h2>Show the last 10 messages</h2>
  <ul id="msg">
  {% for msg in msg_list %}
    <li>({{ msg.when|escape }}) {{ msg.who|escape }} says "{{ msg.what|escape }}"</li>
  {% endfor %}
  </ul>

the perfect test data

Use this string in every webform for testing: [

<"@%'&_\?/:;,>?? €

] ! Here is a (never complete) list of special characters for some common used components:

Put all characters together and spice it with some special utf-8 characters. I used a random character from unicode.org and the special euro sign. If you find problems try the following developer tools to inspect your requests:

• the grandfather of web developer plugins: firebug
• standard request/response plugin Live HTTP headers
• better sniffer: HTTPfox or tamper data
• for ie developer try out Fiddler.com

If these tools wont help, try more logging in your application or use a transparent proxy (for example Charles Web Debugging Proxy for windows).

Back to the basics: The google app engine for python developers provides an environment for web applications, which fullfills many needs:

• a scalable storage (including big table) with 1Gb storage quota, 12GB of incoming and 116 outgoing GB traffic
• the URLfetch API for access to web services like the google crawler
• scaled memcached (10 GB incoming and 50 GB of outgoing traffic)
• Google login integration
• django-compatible web development

First, the app engine dashboard.

The web application is available via the app engine point in my google “my account” and offers 10 slots for 2nd-level domains under *. appspot.com. On the overview page a chart is displayed, which offered a selector with values-pairs such “response time per request” (1) for example. The usage of important quota values follows in a smaller box (2) below. Since the daily available quotas for small web applications are very high, there is usually nothing important (if it does, you should update your account under the billing point). The detailed quota page shows all API calls in real time for the last 24 hours.

Directly below the diagram of the home page there is the list of “Current Load” with paths (3) which in the last 24 hours produced the greatest cpu load. To the right are shown the mostly happed errors per path (4).

The error-log

Like the Apache web server error.log all entries are displayed in an ascending list (filtered by error level) in the dashboard. For application logging use the python logging module. If you carry on an official and a developer version you have to select the small select box in the upper corner (5) for the non default application log (online at

http://#version#latest.#subdomain#.appspot.com

).

look into the datastore

Even the google big table can be accessed directly from the dashboard. You can view them by select-boxes or GQL (a slimmed-SQL dialect). All change in the web application will appear in real time. Whether it delays under load or are large amounts of data, I will review in the future.

Previous general fault cases

In addition to real programming errors (and then displayed directly to your visitors eg, HTTP status code 500), some interesting mistakes in nature are listed. If the google app engine has a common problem, you can visit the status page of google app engine .

catching python exceptions in the app engine application

A general disadvantage of python compared to strictly declarative languages such as java there are not a defined set of exception per modules/function with syntax checking. In python is the agreement that the general failures of one module are caught with module.Error. Special cases can be read in the documentation of the module.

I found the following errors and will try to give some hints to solve the problem.

Error while fetch: ApplicationError: 5

This error occurs after ~ 5 seconds without results on the fetchURL call and could be very common. Its called “DEADLINE_EXCEEDED”

from google.appengine.api import urlfetch
data = urlfetch.fetch(url_with_long_response_time)

This can be intercepted with a simple try-catch. Slow sources should be mitigated with a memcached pre-call

from google.appengine.api import memcache
from google.appengine.api import urlfetch
def fetchUrl(url):
    data = memcache.get(url)
    if !data:
        try:
            data = urlfetch.fetch(url)
            memcache.set(url, data)
        except urlfetch.Error:
            data = ""
    return data
#fetchUrl

Error while fetch: ApplicationError: 2

Wow – the next well named error which is defined in “google/appengine/api/urlfetch_service_pb.py” as “FETCH_ERROR”. This could happen with the following reasons:

• missing host
• MAX_REDIRECTS is exceeded
• the protocol of the redirected URL is bad or missing.
• normal fetching exception like httplib.error, socket.error, IOError

The other ApplicationError numbers of urlfetch are:

• 1: INVALID_URL
• 3: UNSPECIFIED_ERROR
• 4: RESPONSE_TOO_LARGE

And remind the allowed http limitations. Ports in live apps could be 80, 443, 4443, 8080 – 8089, 8188, 8444, 8990 and allowed http methods are GET, POST, HEAD, PUT and DELETE. Other will raise an UNSPECIFIED_ERROR (id=3).

The ResponseTooLargeError will raised if the response (or request) larger than 1 megabyte. It can be catched and you are happy. Or try the option allow_truncated=True in the fetch-function for truncating the response.

Timeout or Error: An error occurred for the API request datastore …

Also in the runtime environment for the python google app engine some components are not available. This applies particularly to the datastore. There are google app engine datastore error handling is needed.

from google.appengine.ext import db
try:
    for stat in db.GqlQuery("select * from ..."):
        pass
except db.Error, exStr:
    logging.error("Error while reading from db:% s"% exStr)

CapabilityDisabledError

That is a strange error and should result in an general error page in the application. Google offers more information in the python traceback:

Datastore writes are temporarily unavailable. 
Please see http://code.google.com/status/appengine for more information.
Traceback (most recent call last):
...
...
...
CapabilityDisabledError: Datastore writes are temporarily unavailable. 
Please see http://code.google.com/status/appengine for more information.

ZeroDivisionError: float division

If you have nothing (or an error) picked up by urlfetch, this error would be possible. The length of the amount of unclaimed items and calculation with the number 0 can provoke the ZeroDivisonError and this is a simple programming error!

UnicodeDecodeError: ‘utf8’ codec …

I have collected some data by urlfetch, but the encoding of the data was ignored. That is a typical application problem. If you do not define the encoding of the incoming data from urlfetch, for example, this errors occurs. Not directly – at least by saving such data in the datastore.

No handlers matched this URL.

Some one try to call a not declared resource of your application. If all your components are correct linking it would be possible a script kiddie having some fun and can be ignored. If the url should be part of your application edit your app.yaml:

handlers:
- url: /geo_data.js
  script: geo_data.py

- url: /index.html
  static_files: geo_demo.html
  upload: geo_demo.html

ValueError: Unterminated string starting at …

This looks like a syntax error which should not happen in a live application. After checking the module the reason is simple. I got JSON data from a web resource and parsing it directly with simplejson! This can be caught like this:

from simplejson import decoder
try:
    result = decoder.JSONDecoder().decode(self.request.get(url))
catch ValueError:
    logging.warn("Error while parsing the json data of %s." % url)
    result= None

Conclusion

official exceptions documentation

That was just a random collection of the common mistakes that you can fix quickly and easily. Here is a link list of the defined exceptions for the mainly used google python modules:

• google datastore exceptions defined in the google.appengine.api.db module
• memcached comes from danga, but google build a new python implementation: google.appengine.api.memcache module. By calling the client-methods there exists only TypeError and ValueError. For serialization the protocollbuffer module is used which can raise an ProtocolBuffer.ProtocolBufferDecodeError.
• the fetchurl api has only the self-explained InvalidURLError, DownloadError and ResponseTooLargeError defined in the google.appengine.api.urlfetch package.
• Exceptions for sending mails are defined in the google.appengine.api.mail package.
• to manipulate images take note the exception defined in google.appengine.api.images package.

other resources

• join the google app engine group
• Run app engine local on your linux system by Charles Severance
• try the books Programming Google App Engine from Dan Sanderson and Using Google App Engine from Charles Severance
• Google App Engine – Python Development (with Eclipse) – Tutorial by Lars Vogel
• Does GAE make python more popular? by Mitch Pronschinske
• The Google app engine blog
• Building a facebook application in the app engine by wubbahed
• the app engine Cookbook
• and some django books: Python Web Development with Django by Wesley Chun, Web 2.0 Programming with Django and Turbogears by Dana Moore and Pro Django (Expert’s Voice in Web Development) by Marty Alchin

The German lottery currently holds a jackpot of about 30 million Euro. A friend of mine took the bait and participated yesterday. Since he is a software developer, he wrote a small program to get him six random numbers in the range of 1 and 49.

Well, it’s not difficult to write such a programm. The challenge is to do so in little bytes. So I challenge you:

Write a JavaScript function that generates random lotto numbers. This function has to return an array of six different numbers from 1 to 49 (including both) in ascending order. You may use features of ECMA-262 only, that means no Array.contains and stuff. You must not induce global variables.

The function has to look like this

var getRandomLottoNumbers = function () {
    // your implementation here
};

Minify your function using JSMin (level aggressive) and count the bytes between the outer curly braces.

To participate in this challenge, simply post your solution as a comment.

Spoiler warning On the next page I describe the steps I took for my solution. Don’t read if you want to submit your own.

First steps

First attempt. My idea is to generate an array containing the numbers from 1 to 49 and then remove elements until six remain. 117 bytes.

var getRandomLottoNumbers = function () {
    var n = [];
    
    for (var i = 1; i < 50; i++) {
        n.push(i);
    }
    
    while (n.length > 6) {
        n.splice(Math.floor(Math.random() * n.length), 1);
    }
    
    return n;
};

First optimization: n.length is used twice, so store it in a variable. Use i, since that is defined already. 114 bytes.

var getRandomLottoNumbers = function () {
    var n = [];
    
    for (var i = 1; i < 50; i++) {
        n.push(i);
    }
    
    while ((i = n.length) > 6) {
        n.splice(Math.floor(Math.random() * i), 1);
    }
    
    return n;
};

Next step: optimize variable declaration. Saves one var. 112 bytes.

var getRandomLottoNumbers = function () {
    var n = [], i;
    
    for (i = 1; i < 50; i++) {
        n.push(i);
    }
    
    while ((i = n.length) > 6) {
        n.splice(Math.floor(Math.random() * i), 1);
    }
    
    return n;
};

Now I changed the algorithm. I switched from a while-loop to a for-loop, which saves another four bytes: 108 bytes.

var getRandomLottoNumbers = function () {
    var n = [], i;
    
    for (i = 1; i < 50; i++) {
        n.push(i);
    }

    for (i = 49; i > 6; i--) {
        n.splice(Math.floor(Math.random() * i), 1);
    }
    
    return n;
};

Below 100 bytes

Don’t you think, Math.floor(Math.random() * i) could be written shorter? Yes, it can. I accidently stumbled upon a way today. Replace it by Math.random() * i | 0. Saves 10 bytes, we’re down to 98.

var getRandomLottoNumbers = function () {
    var n = [], i;
    
    for (i = 1; i < 50; i++) {
        n.push(i);
    }

    for (i = 49; i > 6; i--) {
        n.splice(i * Math.random() | 0, 1);
    }
    
    return n;
};

Save another byte moving the initializing of i one line up. This leaves an empty first statement in the for-loop, but that’s still valid JS. 97 bytes.

var getRandomLottoNumbers = function () {
    var n = [], i = 1;
    
    for (; i < 50; i++) {
        n.push(i);
    }

    for (i = 49; i > 6; i--) {
        n.splice(i * Math.random() | 0, 1);
    }
    
    return n;
};

Now if the first loop starts at 0 and runs until 49, we don’t have to initialize the second for-loop. Saves two bytes, down to 95.

var getRandomLottoNumbers = function () {
    var n = [], i = 0;
    
    for (; i < 49; i++) {
        n.push(i + 1);
    }

    for (; i > 6; i--) {
        n.splice(i * Math.random() | 0, 1);
    }
    
    return n;
};

Save another four bytes by moving the incrementation and decrementation of i. 91 bytes.

var getRandomLottoNumbers = function () {
    var n = [], i = 0;
    
    for (; ++i < 50;) {
        n.push(i);
    }

    for (; --i > 6;) {
        n.splice(i * Math.random() | 0, 1);
    }
    
    return n;
};

Final solution

Now, Douglas Crockford will hate me for this, but some semicolons are optional in Javascript. Remove them to get my final solution of 88 bytes:

var getRandomLottoNumbers = function () {
    var n = [], i = 0;
    
    for (; ++i < 50;) {
        n.push(i)
    }

    for (; --i > 6;) {
        n.splice(i * Math.random() | 0, 1)
    }
    
    return n
};

Addendum, 09/23/2009 16:02

Aww, curly braces are optional in my case. Needs two semicolons again, but saves four braces. 86 bytes

var getRandomLottoNumbers = function () { 
    var n = [], i = 0;
    
    for (; ++i < 50;) n.push(i);

    for (; --i > 6;) n.splice(i * Math.random() | 0, 1);
    
    return n
};

Minimized it’s this:

var getRandomLottoNumbers = function () { 
var n=[],i=0;for(;++i<50;)n.push(i);for(;--i>6;)n.splice(i*Math.random()|0,1);return n
};

Finding an IP address in text or string with python is a simpler task than in java. Only the regex is not shorter than in the java regex example!

First an example with python: build a RegExp-Object for faster matching and than loop over the result iterator.

import re
logText =  'asdfesgewg 215.2.125.32 alkejo 234 oij8982jldkja.lkjwech . 24.33.125.234 kadfjeladfjeladkj'
bytePattern = "([01]?\d\d?|2[0-4]\d|25[0-5])"
regObj = re.compile("\.".join([bytePattern]*4))
for match in regObj.finditer(logText):
    print match.group()

A regex like

/\d+\.\d+\.\d+\.\d+/

wont work, because there match “999.999.111.000” too. But for the usage in python – that is it! Using a regular expression is more native in python than in java. Or in javascript or in perl or asp.net…

And how to find it with JavaScript?

It looks like the small python example. Build the RegExp-Object for faster matching and a loop for finding all.

var logText = 'asdfesgewg 215.2.125.32 alkejo 234 oij8982jldkja.lkjwech . 24.33.125.234 kadfjeladfjeladkj';
var regObj = new RegExp("(?:(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])");
while (var result = regObj.exec(logText)) {
   alert("Matched: `" + result[0]);
}

For a more detailed example have a look at experts-exchange.com.

the native playground : perl

$txt = "asdfesgewg 215.2.125.32 alkejo 234 oij8982jldkja.lkjwech . 24.33.125.234 kadfjeladfjeladkj";
while( $txt=~/(([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5]))/g) {
  print $1."\n";
}

Some regex basics can be found at trap17.com and more regex examples on perl cookbook.

a more complex script: IP – log file statistic

But only finding an IP address in some irregular text is not a common use-case. An apache logfile is well formatted and the IP part can be found directly.

Do a simply split every line and the first part should be the IP address. To check if the first element match the IP pattern, use a ready function (for the example from the socket module). As addition and much more complexer example I will try to find the TOP10 IPs with the request count from the logfile.

import re, socket
hits = {}

#212.174.187.49 - - [13/Jul/2009:01:06:38 +0200] "GET /index.html HTTP/1.1" 400 335 - "-" "-" "-"
try:
    fp = file("all.log")
    for line in fp:
        elements = re.split("\s+", line)
        try:
            socket.inet_aton(elements[0])
            hits[elements[0]] += 1
        except KeyError:
            hits[elements[0]] = 1
        except socket.error:
            pass # no ip in the starting logline
finally:
    fp.close()

#Sorting the IPs with the hit-count
ipKeys = hits.keys()
ipKeys.sort(lambda a,b: hits[b]-hits[a])
for ip in ipKeys[:10]:
    print "%10dx %s" % (hits[ip], ip)

The result looks like this and runs 0.7 sec for 10.000 lines logfile (on Intel Atom N270).

      1406x 10.72.199.111
      1291x 10.214.141.196
       937x 10.43.81.243
       569x 10.43.205.83
       302x 10.235.116.128
       260x 10.121.239.210
       164x 10.145.232.155
       125x 10.106.120.225
       113x 10.93.210.194
       104x 10.174.153.69

The first block of the real addresses has been replaced with the number 10 for anonymity.

And the same IP script as command line with perl

The python script could made shorter and more ugly. Finding all IPs, sorting it, counting and print the TOP10 IPs.

perl -wlne 'print $1 if /(([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5]))/' all.log |sort|uniq -c | sort -n -r|head -n 10

The perl/shell example needs only 0.2 seconds for the same logfile.

conclusion

I like to use regex and python for getting some statistics, but the usage of some command line unix tool is useful too!

In this part I will introduce the python implementation of the ip to geolocation script. It’s more object oriented and hopefully better to read. In the first part of this article I willdescribe the solution to read http resources and parse the content. The second part is the same like the php version. As conclusion I will compare the results of all five APIs with the data from the cache.

• the tutorial how to use ip to geolocation provider api
• the free usage ip to geolocation aggregator script
• the php stand alone implementation
• the python google appengine implementation

python: asynchonous http requests on google appengine

First I want to build the same multi-url-fetch-function (like in php) but I am using the google appengine. There are no threads allowed and the urllib/httplib modules are masked with the urlfetch module from google. I chose the normal and easy urllib.open call because the google backend works fast. After this was done I found in the updated URLFetch-documentation (since June 18, 2009 or appengine version 1.2.3) the section that said: “To do asynchronous calls you have to use the special modul from the urlfetch modul“. Have fun with the improved example.

from google.appengine.api import urlfetch

class InfoItem(dict):
    '''dict with start reading while __init__ the ipinfodb '''
    def __init__(self, url):
        self.rpc = urlfetch.create_rpc()
        urlfetch.make_fetch_call(self.rpc, url)

    def ready(self):
        '''Check if the async call is ready.
        @return True - if got data after parsing
        '''
        try:
            result = self.rpc.get_result()
        except urlfetch.Error, ex:
            logging.error("Error while fetch: %s" % ex)
            return False

        if result.status_code != 200:
            return False
        return self.parse(result.content)
    #ready
#InfoItem

For easy access the result class is based on a python dict. To check if the api data is filled in the dict call the ready() function. You can build the instances of InfoItem, do something other and then ask the instances with the ready-function, if the data has arrived (if not it will wait). Accessing the values is easy because it’s a dict.

Parsing XML data with python

xml should be parsed with the elementtree modul. Its very fast and simple to use. Using the InfoItem class there are two jobs: building the url to the api by simple adding the ip string and parsing the content.

import xml.etree.ElementTree as etree
from xml.parsers.expat import ExpatError

class IpInfoDbItem(InfoItem):
    '''Simple parsing the content of the IpInfoDP-API'''
    def __init__(self, ip):
        '''Init with the IpInfoDb-url'''
        super(IpInfoDbItem, self).__init__("http://ipinfodb.com/ip_query.php?ip="+ip)

    def parse(self, content):
        '''Parse the IpInfoDb-XML and save the keys in the inner dict.
        @return True - if parsing was successfull.
        '''
        try:
            #etree needs a file-like-object instead a string!
            t = etree.ElementTree().parse(StringIO.StringIO(content))
            self.update({'name': 'ipinfodb',
                         'country': t.find("CountryName").text or '',
                         'city':    t.find("City").text or '',
                         'lat':     float(t.find("Latitude").text),
                         'long':    float(t.find("Longitude").text)})
            return True
        except (ExpatError, IOError), ex:
            logging.warn("Nothing parsed: %s" % ex)
        return False
    #parse
#IpInfoDbItem

#Test the code directly (if google modules are in the path)
testing = IpInfoDbItem("127.0.0.1")
if testing.ready():
    print testing
#  {'lat': 0.0, 'country': 'Reserved', 'name': 'ipinfodb', 'long': 0.0, 'city': None}

The example starts fetching the data from the IpInfoDb-API in the __init__ function, parses the xml und fills the values in the dict with self.update.

Parsing non-structured data with python

The same hint like in php – use regular expressing for matching the data!

import re

class HostIpItem(InfoItem):
    '''dict with reading while __init__ the hostip '''
    def __init__(self, ip):
        super(HostIpItem, self).__init__("http://api.hostip.info/get_html.php?position=true&ip="+ip)

    def parse(self, content):
        '''Parse the HostIp-Text and save the keys in the inner dict.
        @return True if parsing was successfull.
        '''
        match = re.search("Country:\s+(.*?)\(\w+\)\nCity:\s+(.*?)\nLatitude: (-*\d+\.\d+)\nLongitude: (-*\d+\.\d+)", content, re.S|re.I)
        if match:
            self.update( {'name': 'hostip',
                          'country': match.group(1),
                          'city': match.group(2),
                          'long': float(match.group(4)),
                          'lat': float(match.group(3))})
            return True
        return False
    #parse
#HostIpItem

Works like the xml example …

Build a complete webapplication

To put this together you have to define a RequestHandler, who fetches the data and produces a javascript. In django style you need the following template, the values in {{ x }} will be replaced with a dict.

var com = com||{};
com.unitedCoders = com.unitedCoders||{};
com.unitedCoders.geo = com.unitedCoders.geo||{};
com.unitedCoders.geo.ll = {{ ll_json }} ;

{{ maxmind }}
{{ wipmania }}
{{ google }}
document.write('<script type="text/javascript" src="http://pyUnitedCoders.appspot.com/geo_func.js"></script>');

com.unitedCoders.geo.staticMapUrl = function(x, y) {
  var url = "http://maps.google.com/staticmap?key={{ google_key }}&size="+x+"x"+y+"&markers=";
  var colors = ["blue","green","red","yellow","white", "black"];
  for (var i=0; i<com.unitedCoders.geo.ll.length;i++) {
      var s = com.unitedCoders.geo.ll[i];
      url += s.lat+","+s.long+",mid"+colors[i]+(i+1)+"%7C";
  };
  url += this.getLat() + ","+this.getLong() + ",black";
  return url;
};

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.ext.webapp import template

class GeoScript(webapp.RequestHandler):
    def get(self):
        '''Get the location infos for the calling ip (from api).'''

        self.response.headers['Content-Type'] = 'text/plain;charset=UTF-8'

        #result-dict and local location list
        result = {}
        ll = []

        #Start fetching API data
        ipInfo = IpInfoDbItem(ip)
        hostIp = HostIpItem(ip)

        #Add some more Javascrip APIs
        scriptTemp = "document.write('<script type=\"text/javascript\" src=\"%s\"></script>');"
        result['maxmind'] = scriptTemp % "http://j.maxmind.com/app/geoip.js"
        result['wipmania'] = scriptTemp % "http://api.wipmania.com/wip.js"
        if self.request.get("key"):
            result['google_key'] = self.request.get("key")
            result['google'] = scriptTemp % \
                      ("http://www.google.com/jsapi?key=" + self.request.get("key"))

        #Get the fetched API Data
        if ipInfo.ready():
            ll.append(ipInfo)
        if hostIp.ready():
            ll.append(hostIp)

        result['ll_json'] = encoder.JSONEncoder().encode(result['ll'])

        #Put all together in the javascript template
        path = os.path.join(os.path.dirname(__file__), 'geo.temp')
        self.response.out.write(template.render(path, result))
    #get
#GeoScript

application = webapp.WSGIApplication([
        ('/geo_data.js', GeoScript) ], debug=True)

For more information on how to start a python google appengine Webapplication start reading the fine google documentation!

conclusion

Don't mix too many languages - you will be confused! The parallel implementation of the server side script in php and python and using one version for the advanced functions in javascript will mix three script language! My first failures have been setting some semicolons in python or forgetting the block parentheses in javascript.

After deploying and watching a server side script with the dashboard of google's appengine you get all data: Logs and API-calls in detail, you can manage many different versions (default is one) or give deploy access to other google accounts:That's great!

What is the best service provider?

I have done some caching and checked all five API results. Here is the hit rate for the location of visitors of this blog and the distance to the center from the given locations (The center is the average of all long/lat values pairs with a given city value per ip).

HostIP and google do not offer location data for many visitors. IPInfoDB and MaxMind do not have the same positions (like suggested in the comments) for all IPs. At this time WIPMania mainly offers the center of the country. So the positions are not very accurate (in comparision to the calculated center).

How calculate the distance between lat/long values?

I found some nice functions in javascript (please don't add functions to the prototype to String and Integer!!!), in python the distance function looks like the following lines:

def distance(lat1, long1, lat2, long2):
    return 6378.7 * math.acos(math.sin(lat1/57.2958) * math.sin(lat2/57.2958) + math.cos(lat1/57.2958) * math.cos(lat2/57.2958) * math.cos(lon2/57.2958 - lon1/57.2958))
#distance

The implementation of the ip to geo location script started as a quick hack in php. Sometimes I use php scripts because everyone can include it in his own website account (but I am not an active php programmer). It’s much easier to write a simple php script instead of deploying a more complex web application. The quick result was this implementation as an example. After getting the php script to work I rewrote it in python, transformed it into a google appengine application and deployed it. Both worked fine and now there are two examples for the API integration available.

• the tutorial how to use ip to geolocation provider api
• the free usage ip to geolocation aggregator script
• the php stand alone implementation
• the python google appengine implementation

php and http API / RESTfull web services

To fetch http resources use the the curl module. It’s robust and there aren’t too many possibilities to break it. If you want to use more than one service provider’s API you can access them in a parallel fashion. So the total execution time for the request will be equal to the slowest provider’s roundtrip time instead of the sum of all roundtrip times together.

  function readUrls($urls) {
    $req = array();
    $mh = curl_multi_init();
    for($i = 0; $i<pre class="prettyprint"> 0);

    //read the content
    $ret = array();
    for($i = 0; $i<count($urls); $i++) {
      $ret[$i] = curl_multi_getcontent($req[$i]);
      curl_multi_remove_handle($mh, $req[$i]);
    }
    curl_multi_close($mh);
    return $ret;
  }

This simple function gets an array of urls and returns an array of the content. Ok - only the optimistic way without error detection!

Basic XML parsing with PHP

First you have to check if the Content-encoding is set correctly. If not you'll have to convert the http-content to the encoding declared within the xml declaration, which is UTF-8 in the example below.

<?xml version="1.0" encoding="UTF-8"?>
<Response>
        <Ip>74.125.45.100</Ip>
        <Status>OK</Status>
        <CountryCode>US</CountryCode>
        <CountryName>United States</CountryName>
        <RegionCode>06</RegionCode>
        <RegionName>California</RegionName>
        <City>Mountain View</City>
        <ZipPostalCode>94043</ZipPostalCode>
        <Latitude>37.4192</Latitude>
        <Longitude>-122.057</Longitude>
</Response>

And now the simple php code for xml parsing:

if (substr($content,1,5)=='?xml') {
    $xml = simplexml_load_string(utf8_encode($content));
    if ($xml && $xml->City) {
    //Add the geo data to the result set
    $result =  array("name" => "ipinfodb",
                      "country" => strval($xml->CountryName),
                      "city" => strval($xml->City),
                      "long" => floatval($xml->Longitude),
                      "lat" => floatval($xml->Latitude)  );
}

php and parsing non-structured data

For non-structured data regular expressions are the best solution if you can beat the beast. Otherwise try it with several split-actions.

Country: UNITED STATES (US)
City: Sugar Grove, IL
Latitude: 41.7696
Longitude: -88.4588

And now the php code for text parsing:

if (preg_match("/Country:\s+(.*?)\(\w+\)\nCity:\s+(.*?)\nLatitude: (-*\d+\.\d+)\nLongitude: (-*\d+\.\d+)/", $content, $ma)==1) {
    $result = array("name" => "hostip",
                              "country" => $ma[1],
                              "city" => $ma[2],
                              "long" => floatval($ma[4]),
                              "lat" => floatval($ma[3])  
};

php and parsing JSON data

Json is defined as the JavaScript object notation and is an universal key-value transfer format, especially in web applications. So many API (like the IPInfoDB from the xml example) offer these formats and php has the build-in function to parse it. The encoding is defined as utf-8!

if (substr($content,0,1)=='{') {
    $json =  json_decode($content);
    if ($json && $json->{'City'}) {
        $result = array("name" => "ipinfodb",
                                  "country" => $json->{'CountryName'},
                                  "city" => $json->{'City'},    
                                  "long" => floatval($json->{'Longitude'}),
                                  "lat" => floatval($json->{'Latitude'})));
    }
}

Getting the javascripts data together

The described php code should produce a simple javascript (after running the above script) containing the fetched data and including the other javascripts via document.write function like this output:

var com = com||{};
com.unitedCoders = com.unitedCoders||{};
com.unitedCoders.geo = com.unitedCoders.geo||{};
<?php
  //build your result
  echo "com.unitedCoders.geo.ll = [".json_encode($result)."];\n";
?>
document.write('<script type="text/javascript" src="http://j.maxmind.com/app/geoip.js"></script>');
document.write('<script type="text/javascript" src="http://api.wipmania.com/wip.js"></script>');
document.write('<script type="text/javascript" src="geo.js"></script>');

After the other javascript sources are loaded (sequential) the geo_funcs.js javascript can map from the internal structures to the global com.unitedCoders.geo.ll list and add some helper functions. For example the "get the middle position" of all lat/long values.

add the external javascript data to the result

The maxmind javascript api has as result a set of global functions. Here the maxmind part of geo_funcs.js:

if (window['geoip_country_name']) {
    com.unitedCoders.geo.ll.push({
            "name": "maxmind",
            "country": geoip_country_name(),
            "city": geoip_city(),
            "long": parseFloat(geoip_longitude()),
            "lat": parseFloat(geoip_latitude()) });
};

//calculate the middle point of all position where city is set
(function() {
    var lat = 0.0;
    var long = 0.0;
    var count = 0;
    for (var i=0; i<com.unitedCoders.geo.ll.length;i++) {
        var service=com.unitedCoders.geo.ll[i]
        if (service.long && service.lat && service.city) {
            lat += parseFloat(service.lat);
            long += parseFloat(service.long);
            count ++;
       }
    }
    com.unitedCoders.geo.lat = lat/count;
    com.unitedCoders.geo.long = long/count;
})();

The function wrapper chooses all positions with a set city, hoping these lat/long value pairs are the most accurate positions and calculate the middle point. After testing I noticed the google positions are accurat (but no city information is available).

In the next article I will describe the same functionality but implemented in python for the google appengine.

If you provide a restaurant guide it would be great to show the next restaurant based on webpage visitors position or the local beer garden specials if its sunny weather? Or offer geo targeted ads on your page? And this could be offered without registration or connection to a social network? I will describe the api/implementation details and offer directly our free “ip to geolocation aggregator”-script.

The first step is determining geo location based on Internet IP address of the visitor. I have found five free service providers which offer data based on the client-IP the geo position and city/country data. Classic “ip to geolocation data offers” are commercial. You have to buy and download a sql dump or csv file with ip ranges from isp with country and city data – some expanded with long/lat values. The second solution (and to save hosting space) is using a http api which offer theses data directly when being called – these looks like the preferred possibility. And the third way is to include a javascript which can integrate the geo position directly without need of a server component.

Hostip.info

HostIP is a community-powered database of IP mapping with a simple REST API. But you can also get the database dump if you have a local database and want to integrate . Simple start a HTTP request with the IP of your visitor and get the following 4-line-result:

• URL:

  http://api.hostip.info/get_html.php?ip=xxx&position=true

• Country: UNITED STATES (US)
  City: Sugar Grove, IL
  Latitude: 41.7696
  Longitude: -88.4588
  

For simple integration I will use php lines, php output will be javascript: Fetch the data with the curl module and parse it with a simple regex!

preg_match("/Country:\s+(.*?)\nCity:\s+(.*?)\nLatitude: (\d+\.\d+)\nLongitude: (\d+\.\d+)/", $DATA, $ma);
echo "  hostip:{country: '".$ma[1]."', city: '".$ma[2]."', long: ".$ma[4].", lat:".$ma[3]."}";

IPInfoDB.com

At ipinfodb (or better known as iplocationtools.com, found via programmableweb.com) you can find some free service to get the position for an IP via API webservice (limitation is 500 connections at a rate faster than 3 per second during a 1h period) or all the data as a sql dump. I choose the API webservice (easy to integrate) with XML-Output, because the city names are save with xml and utf-8 encoding!

• URL:

  http://iplocationtools.com/ip_query.php?ip=xxx

• <?xml version="1.0" encoding="UTF-8"?>
  <Response>
  	<Ip>74.125.45.100</Ip>
  	<Status>OK</Status>
  	<CountryCode>US</CountryCode>
  	<CountryName>United States</CountryName>
  	<RegionCode>06</RegionCode>
  	<RegionName>California</RegionName>
  	<City>Mountain View</City>
  	<ZipPostalCode>94043</ZipPostalCode>
  	<Latitude>37.4192</Latitude>
  	<Longitude>-122.057</Longitude>
  </Response>
  

What do to? Fetch the data and use the simple xml parser to fetch the attributes. php output will be javascript.

$xml = simplexml_load_string($DATA);
echo "  iplocationtools: {country: '".$xml->CountryName."', city: '".$xml->City."', long: ".$xml->Longitude.", lat:".$xml->Latitude."}";

MaxMind.com

MaxMind.com doesn’t offer a free Webservice like the others. You have to download a binary package with all IP related data for your preferred language. That isn’t as convenient as calling an REST interface.

Our iplocation script will include the javascript url as dynamic loaded javascript. After loading the dynamic parts it looks in the global javascript context for the maxmind functions and include the values in the result object.

• URL: http://j.maxmind.com/app/geoip.js

• function geoip_country_code() { return 'DE'; }
  function geoip_country_name() { return 'Germany'; }
  function geoip_city() { return 'Stuttgart'; }
  function geoip_region() { return '01'; }
  function geoip_region_name() { return 'Baden-Württemberg'; }
  function geoip_latitude() { return '48.7667'; }
  function geoip_longitude() { return '9.1833'; }
  function geoip_postal_code() { return ''; }
  

This javascript interface is ugly and the data defing global functions, which can’t be removed from the global namespace.

<script type="text/javascript" src="http://j.maxmind.com/app/geoip.js"></script>
<script type="text/javascript"> 
if (window.geoip_country_name) {
  com.unitedCoders.geo.ll.push({
      'name':    'maxmind', 
      'country': geoip_country_name(),
      'city':  geoip_city(),
      'long':  parseFloat(geoip_longitude()), 
      'lat':   parseFloat(geoip_latitude())    });
};
</script>

Here the Javascript code to add the maxmind values in the locationlist of our the global javascript object.

Google AJAX API

The google AJAX API offers a basic loader for many popular javascript libs. I found a nice function at startup: a global google.loader.ClientLocation – object. After some tests the city parameter is not as accurate as in the other location services, but the lat/long values are fine!

<script  type="text/javascript" src="http://www.google.com/jsapi?key=###api-key###"></script>
<script  type="text/javascript">
  if (window['google'] && google.loader.ClientLocation) {
    com.unitedCoders.geo.ll.push({
      'name':    'google', 
      'country': google.loader.ClientLocation.country,
      'city':       google.loader.ClientLocation.city,
      'long':      google.loader.ClientLocation.longitude, 
      'lat':        google.loader.ClientLocation.latitude    });
  };
</script>

To include the google parameters the google api key must be generated for the domain where the javascript will be used. For this tutorial I used unitedcoderscom.appspot.com as the domain.

WIPmania Location Service

And WIPmania offers a javascript API based on a global javascript object, integration is the same as with the google api.

<script type="text/javascript" src="http://api.wipmania.com/wip.js"></script>
<script  type="text/javascript">
if (window['WIPlocation']) {
   com.unitedCoders.geo.ll.push({
           'name':    'WIPmania',
           'country': WIPlocation.address.country,
           'city':  WIPlocation.address.city,
           'long':  WIPlocation.longitude,
           'lat':   WIPlocation.latitude    });
};
</script>

The same integration like the google api but no need for a application key! But after some tests it turned out that the location data is not too precise!

Geo Location API in the browser

After some googling I found a hint from Adam to get the geo position directly via javascript if the browser is running on a mobile device. The iphone implementation is available with the OS 3.0 update since June 17th 2009. In the future there are also plans to include this with a official W3C API that differs only in the position object. Both implementations provide only the lat/long values not the nearest city’s name. Including looks simple like the other javascript services described below:

if (window['navigator'] && navigator['geolocation']) {
  navigator.geolocation.getCurrentPosition(function(pos) {
    if (pos['coords']) {
      com.unitedCoders.geo.ll.push({
              name: 'w3c geo-api',
              long: pos.coords.longitude,
              lat: pos.coords.latitude });
    } else {
      com.unitedCoders.geo.ll.push({
              name: 'safari on iphone',
              long: pos.longitude,
              lat: pos.latitude });
    };
  });
};

An automatically integration is not useful because there starts a ugly popup dialog to ask the user if the location service should be available for the safari application and then for my domain.

Combine and Improve it

Every IP2geolocation service provider maintain his own database. I got at home three different towns with coordinates that differs for 50 kilometers. To improve this you can choose the two best positions (and discard the rest) or calculate the center point of all results. For some countries the geo location providers knows only the center point of the country so I have chosen only locations with a given city and calculated the middle point of these records for the lat/long values. These values will come with the lat/long-functions.

var com.unitedCoders.geo = {
  ll: [ { name: 'maxmind', country: 'Germany', city: 'Stuttgart', long: 9.1833, lat: 48.7667}, 
        { name: 'iplocationtools', country: 'Germany', city: 'Bietigheim-bissingen', long: 9.1333, lat:48.9667},
        { hostip: {country: 'GERMANY (DE)', city: 'Karlsruhe', long: 8.4, lat:49.05},
      ],
  getLong: function(), //8.905533333333333,
  getLat: function() // 48.9278
};

After that I have hopefully a more precise position of the visitor than using only one geo location provider’s api. Now I need some services to get content based on this information. For integration of the script I choose to display the position on a map as a simple example.

Show the google map

To visualize the positions on a map the full featured googlemap is too big – I am prefering the static google map. It is just a dynamically build img-url with some parameters (get for your domain a google map api key) which returns a static image.

• URL: http://maps.google.com/staticmap?size=400×300&markers={latitude},{longitude},{size}{color}{alphanumeric-character}

If nothing other given the map will center all markers and fit the zoom level. The API is limited to 1000 calls/hour (of different visitors) but google will be nice in the limits.

And now : I know where you are!

How to integrate all these geo location API?

In Part 2 I will explain with a short php script and a little python script (in the google app engine) the implementation details. For free usage you can include the ip2geoLocation aggregator with the simple integration via one javascript.

If you find more free ip-to-geolocation service proiver feel free to comment this article so we can try to integration the service.