Scaling Apps on the Google App Engine

When I first became aware of Google App Engine, I was sceptical. It seemed to have some severe shortcomings that would make it unable to match up to competitors, especially Amazon Web Services.

These perceived shortcomings included the fact that your Google applications do not have access to an entire virtual server setup in the way that, for example, Amazon.com’s Amazon EC2 (Elastic Compute Cloud) does. On EC2, applications get an entire virtual private server, complete with an entire operating system of your choice (including Linux, Microsoft Windows Server 2003 and even Sun Microsystems’ Solaris). The applications can then be written and compiled in any language supported by your chosen operating system, and run on the virtual server.

Google App Engine, on the other hand, limits you to a tightly controlled Python virtual machine. Your applications can only be written in Python, and you don’t have access to a file system or any of the operating system features. And Python isn’t a truly compiled language.

Furthermore, unlike competitors’ offerings, your Google applications don’t have access to a full SQL-based database. Instead, if you want to use a database, you must use the Google Datastore.

As it happens, I was not the only one with these concerns. Many bloggers and reporters wrote about these issues. In response, many people came to Google’s defense and offered explanations of why these were, in fact, not shortcomings at all, but rather just a different approach to a common problem of creating software that can easily scale and handle millions of users.

People could argue either side and present different ideas about whether Google’s approach is good. However, without any real evidence, it was hard to back up either side of the debate. That said, there was one strong argument indicating the plan would work: Google’s own products. It’s no secret that much of the Google App Engine, or GAE, is based on the very same technology that powered Google’s own products, including its search engine. For example, the Datastore was built using technology very similar to the database technology used behind Google Maps, Blogger and even the search engine itself.

Now, almost a year later, the evidence is starting to come in, although slowly. In this article I explore a few applications that developers have built with GAE, and then draw some conclusions about the current state of GAE.

Google itself provides a list of some “Editor’s Picks” for applications. These are what people at Google consider among the best applications they’ve seen. For me, that’s a good starting point, because such applications should (hopefully) show what the engine is capable of. You can see the list at the Application Gallery site.