Back to the basics: The google app engine for python developers provides an environment for web applications, which fullfills many needs:
- a scalable storage (including big table) with 1Gb storage quota, 12GB of incoming and 116 outgoing GB traffic
- the URLfetch API for access to web services like the google crawler
- scaled memcached (10 GB incoming and 50 GB of outgoing traffic)
- Google login integration
- django-compatible web development
First, the app engine dashboard.
The web application is available via the app engine point in my google “my account” and offers 10 slots for 2nd-level domains under *. appspot.com. On the overview page a chart is displayed, which offered a selector with values-pairs such “response time per request” (1) for example. The usage of important quota values follows in a smaller box (2) below. Since the daily available quotas for small web applications are very high, there is usually nothing important (if it does, you should update your account under the billing point). The detailed quota page shows all API calls in real time for the last 24 hours.
Directly below the diagram of the home page there is the list of “Current Load” with paths (3) which in the last 24 hours produced the greatest cpu load. To the right are shown the mostly happed errors per path (4).
Like the Apache web server error.log all entries are displayed in an ascending list (filtered by error level) in the dashboard. For application logging use the python logging module. If you carry on an official and a developer version you have to select the small select box in the upper corner (5) for the non default application log (online at
look into the datastore
Even the google big table can be accessed directly from the dashboard. You can view them by select-boxes or GQL (a slimmed-SQL dialect). All change in the web application will appear in real time. Whether it delays under load or are large amounts of data, I will review in the future.
Previous general fault cases
In addition to real programming errors (and then displayed directly to your visitors eg, HTTP status code 500), some interesting mistakes in nature are listed. If the google app engine has a common problem, you can visit the status page of google app engine .
catching python exceptions in the app engine application
A general disadvantage of python compared to strictly declarative languages such as java there are not a defined set of exception per modules/function with syntax checking. In python is the agreement that the general failures of one module are caught with module.Error. Special cases can be read in the documentation of the module.
I found the following errors and will try to give some hints to solve the problem.
Error while fetch: ApplicationError: 5
This error occurs after ~ 5 seconds without results on the fetchURL call and could be very common. Its called “DEADLINE_EXCEEDED”
from google.appengine.api import urlfetch data = urlfetch.fetch(url_with_long_response_time)
This can be intercepted with a simple try-catch. Slow sources should be mitigated with a memcached pre-call
from google.appengine.api import memcache from google.appengine.api import urlfetch def fetchUrl(url): data = memcache.get(url) if !data: try: data = urlfetch.fetch(url) memcache.set(url, data) except urlfetch.Error: data = "" return data #fetchUrl
Error while fetch: ApplicationError: 2
Wow – the next well named error which is defined in “google/appengine/api/urlfetch_service_pb.py” as “FETCH_ERROR”. This could happen with the following reasons:
- missing host
- MAX_REDIRECTS is exceeded
- the protocol of the redirected URL is bad or missing.
- normal fetching exception like httplib.error, socket.error, IOError
The other ApplicationError numbers of urlfetch are:
- 1: INVALID_URL
- 3: UNSPECIFIED_ERROR
- 4: RESPONSE_TOO_LARGE
And remind the allowed http limitations. Ports in live apps could be 80, 443, 4443, 8080 – 8089, 8188, 8444, 8990 and allowed http methods are GET, POST, HEAD, PUT and DELETE. Other will raise an UNSPECIFIED_ERROR (id=3).
The ResponseTooLargeError will raised if the response (or request) larger than 1 megabyte. It can be catched and you are happy. Or try the option allow_truncated=True in the fetch-function for truncating the response.
Timeout or Error: An error occurred for the API request datastore …
Also in the runtime environment for the python google app engine some components are not available. This applies particularly to the datastore. There are google app engine datastore error handling is needed.
from google.appengine.ext import db try: for stat in db.GqlQuery("select * from ..."): pass except db.Error, exStr: logging.error("Error while reading from db:% s"% exStr)
That is a strange error and should result in an general error page in the application. Google offers more information in the python traceback:
Datastore writes are temporarily unavailable. Please see http://code.google.com/status/appengine for more information. Traceback (most recent call last): ... ... ... CapabilityDisabledError: Datastore writes are temporarily unavailable. Please see http://code.google.com/status/appengine for more information.
ZeroDivisionError: float division
If you have nothing (or an error) picked up by urlfetch, this error would be possible. The length of the amount of unclaimed items and calculation with the number 0 can provoke the ZeroDivisonError and this is a simple programming error!
UnicodeDecodeError: ‘utf8’ codec …
I have collected some data by urlfetch, but the encoding of the data was ignored. That is a typical application problem. If you do not define the encoding of the incoming data from urlfetch, for example, this errors occurs. Not directly – at least by saving such data in the datastore.
No handlers matched this URL.
Some one try to call a not declared resource of your application. If all your components are correct linking it would be possible a script kiddie having some fun and can be ignored. If the url should be part of your application edit your app.yaml:
handlers: - url: /geo_data.js script: geo_data.py - url: /index.html static_files: geo_demo.html upload: geo_demo.html
ValueError: Unterminated string starting at …
This looks like a syntax error which should not happen in a live application. After checking the module the reason is simple. I got JSON data from a web resource and parsing it directly with simplejson! This can be caught like this:
from simplejson import decoder try: result = decoder.JSONDecoder().decode(self.request.get(url)) catch ValueError: logging.warn("Error while parsing the json data of %s." % url) result= None
official exceptions documentation
That was just a random collection of the common mistakes that you can fix quickly and easily. Here is a link list of the defined exceptions for the mainly used google python modules:
- google datastore exceptions defined in the google.appengine.api.db module
- memcached comes from danga, but google build a new python implementation: google.appengine.api.memcache module. By calling the client-methods there exists only TypeError and ValueError. For serialization the protocollbuffer module is used which can raise an ProtocolBuffer.ProtocolBufferDecodeError.
- the fetchurl api has only the self-explained InvalidURLError, DownloadError and ResponseTooLargeError defined in the google.appengine.api.urlfetch package.
- Exceptions for sending mails are defined in the google.appengine.api.mail package.
- to manipulate images take note the exception defined in google.appengine.api.images package.
- join the google app engine group
- Run app engine local on your linux system by Charles Severance
- try the books Programming Google App Engine from Dan Sanderson and Using Google App Engine from Charles Severance
- Google App Engine – Python Development (with Eclipse) – Tutorial by Lars Vogel
- Does GAE make python more popular? by Mitch Pronschinske
- The Google app engine blog
- Building a facebook application in the app engine by wubbahed
- the app engine Cookbook
- and some django books: Python Web Development with Django by Wesley Chun, Web 2.0 Programming with Django and Turbogears by Dana Moore and Pro Django (Expert’s Voice in Web Development) by Marty Alchin