Castles in the Cloud:Developing with Google App EngineKevin NoonanCalbane Ltd.IT@Cork, 20th April, 2009

Introduction

Castles in the Cloud

A working definition of Cloudware

• Outsourced Virtualizationo Virtual server running on outsourced server farm

The Cloud Computing Pyramid

•Michael Sheehan of GoGrid devised this classification

Main Players

Other Players

• Microsoft Windows Azureo Including .NET Live Services

• Force.com from Salesforce

• SUN (recently announced)

• Mosso

• GoGrid

• FlexiScale

• Hosting365

• ...

What's Cloudware good for? (I)‏

• Guerilla developmento Pay-per-use model is ideal for startupso Also good for guerillas inside the enterprise...

What's Cloudware good for? (II)‏

• Scaling upo Clone the virtual servero pay for extra resources (bandwidth, storage, RAM, etc.)‏

What's Cloudware good for? (III)‏

• In general, outsourcing IT infrastructureo pay someone else to run your servers, network, database...

Who's using Cloudware & why not?Castles in the Cloud

• Startupso Twittero SmugMugo Pixenate (Walter Higgin's site, from here in Ireland)‏o Many Facebook apps, e.g. iLIke

• Large firmso NASDAQo Activisiono Business Objectso Hasbroo New York Times

• See: http://aws.amazon.com/solutions/case-studies/

Startups & Large Firms on AWS

• Startupso BuddyPoke! (30M daily pageviews on App Engine)‏o Pixverse (PixChat)‏o Hyperic (Cloudstatus)‏

• One large company's experiment:o GiftTag from BestBuy

• http://code.google.com/appengine/casestudies.html

Startups on Google App Engine

• Immaturity of the technologyo Although AWS is over three years old

• Concerns about reliability

• Concerns about portabilityo no two clouds are the same

• Bureaucracy & inertia

• Lock-in to legacy systems

• Fear of loss-of-control & of job-losses

• ...

Barriers to Enterprise Adoption

• Amazon S3 went down for two consecutive hours in Feb 2008.

• Amazon S3 (& Amazon SQS) went down for eight consecutive US daytime hours in

July 2008.

• App Engine had a five-hour partial-outage in June '08

• App Engine had a problem of increased latency (for web-serving) over a few days in

early Mar '09.

• Group for notification of App Engine downtimeo http://groups.google.com/group/google-appengine-downtime-notify

Black days in the Cloud

Google App EngineCastles in the Cloud

Python: The Original of the Species

Java: the All New Flavour

Languages on App Engine's JVM

• Java

• JRuby

• Jython

• BeanShell

• JavaScript

• Scala

• Groovy

• …

• Will it Play in App Engine?

http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine

What is Google App Engine?• Complete web stack

o Soon to be more general computing platform

• Runs on Google's infrastructure

• Bigtable Datastoreo Like a sorted hashtableo Not a relational databaseo Use GQL (with Python) instead of SQLo Use JDO or JPA (with Java) for datastore work

• Memcache

• Image API

• Mail API

• Python library available for Force.com web services

• SDK for development on Windows/Mac/Linuxo Run & test locally before uploading to Cloud

A fabulous app: 'Gaebar'

Alternative: use AppRocket to replicate an App Engine datastore to an SQL database

http://code.google.com/p/approcket/

Limitations of Google App Engine• Applications run in “The Sandbox”:

http://code.google.com/appengine/docs/java/runtime.html#The_Sandbox

• Native Python modules (written in C) do not work.

• Many Java libraries will not work (but many have been patched or will be).

See the JRE whitelist: http://code.google.com/appengine/docs/java/jrewhitelist.html

• No “long-lived” processing (requests must return in 30 seconds).

• Limited “offline processing” (this is coming)o no forks, no spawned processes, no threadso Just scheduled tasks, which work like “cron jobs” on UNIX systems

• HTTPS only on XYZ.appspot.como and only with Google's certificate.o Due to limitation of SSL protocol, as applied to Google's Cloud.

Roadmap to June for App Engine• Service for storing and serving large files

• Task queues for performing background processing

• Incoming email support

• XMPP (Jabber) API

'Batch processing' on App Engine today• Use App Engine's task scheduling for “cron jobs” spaced a minute apart (or more seldom)

• A hack for more frequent jobs: run a cron-job on your PC or server to periodically hit a URL on App

Engine and kick off a task

• An outsourced hack for more frequent jobs: use webcron.org o do the same as above (perhaps more reliably!) o Cost: one hundredth of one Euro cent per 'hit'

• More sophisticated: hitch the clouds togethero 'App Engine Patch' has support for 'Boto'o 'Boto' is a Python library for Amazono Use Boto (on App Engine) to queue work-packets on Amazon's Simple Queue Service (SQS) o Kick off tasks on App Engine with a method above.o Cost: $0.01 per 10,000 requestso (+ $0.10 per GB transferred in...)

Web tools for App Engine

• “Xmas gifts” - announced in late December 2008

• Zoho Creatoro generates code and web-pages for deploymento http://creator.zoho.com/

• App Engine Site Creatoro “Lightweight CMS” (with GUI interface)‏o http://code.google.com/p/app-engine-site-creator/o Apache 2.0 license

Billing on Google App Engine• True pay-per-useo free quotas & no bill to pay if quotas not exceeded.o quite different model to AWS

• Developer can assign budget per resource.

• Priceso $0.10 per CPU core-houro $0.15 per GB-month of storageo $0.12 per GB outgoing bandwidtho $0.10 per GB incoming bandwidtho $0.0001 per email recipient for emails sent by the application

Free Quotas on App Engine

•Daily outgoing bandwidth: 10 GB

•Daily incoming bandwidth 10 GB

•Daily requests 1.3 million

• Daily CPU-hours: 46

• More quotas (mail, datastore, images, memcache)

http://code.google.com/appengine/docs/quotas.html

Other limitations on App Engine

• 1 GB of free storage per app.

•100 MB Memcache storage limit

• App Engine apps can now receive requests and send responses of up to 10MB.

• Caveat: 1 MB limit on datastore/memcache API call

• You can deploy up to 1,000 code and 1,000 static files with an app.

Workaround: use ZipImport (with Python) to store files in a Zip-archive:

http://code.google.com/appengine/articles/django10_zipimport.html

• 150MB cumulative limit on an app's code and static files

(as opposed to storage in the datastore).

Atypical Uses of App Engine

• Run Open Source software (e.g. blog / CMS):

http:/groups.google.com/group/google-appengine/web/google-app-engine-open-source-projects

• Store your static files on App Engineo You can use App Engine as rudimentary 'Content Delivery Network'

• Expose a web-service on App Engine (while doing your main development elsewhere)

REST library for Django & App Engine: http://code.google.com/p/python-rest/

• Expose an interface to BigTable and exploit its scalable storage:

REST interface to datastore: http://code.google.com/p/app3/

• Use RESTlets for Java equivalent of the two above:

http://wiki.restlet.org/docs_1.2/13-restlet/252-restlet.html

Brief Case Study: LongURLPlease

• LongURLPlease expands “tiny URLs” to the original long versions.

• Original version written in a day by Darragh Curran, with barely any previous Python experience.

• Launched publicly on December 23rd, 2008.

• Rapidly became a platform in itself: at least nine other applications built on the API within 8 weeks

• Over four thousand downloads of the Firefox plugin

• 100K – 200K requests per day (each request can batch up to 4 URLs)

• Running costs: zero (since its usage is well below the free quotas)

Where to go from here (Python)

• App Engine Helper tutorialo http://blog.pas.net.au/2009/02/17/django-and-google-app-engine-tutorial/

• App Engine Patch article (I recommend AEP over the “helper” above)o http://code.google.com/appengine/articles/app-engine-patch.html

• App Engine cheatsheet: http://short.ie/cheatongae

• Code samples o http://code.google.com/p/google-app-engine-samples/downloads/list

Where to go from here (Java)

• Sign up for the Java “early look”

http://appengine.google.com/promo/java_runtime

• Install the Eclipse plugin (& SDK)

http://code.google.com/appengine/docs/java/tools/eclipse.html

(The Java samples are in the SDK.)

• Read Google's docs and follow the tutorial:

http://code.google.com/appengine/docs/java/overview.html

• Or pick your own language and get up-and-running on the JVM:

http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine

2/27/2009

ReferenceCastles in the Cloud

• 10 Reasons Enterprises Aren't Ready to Trust the Cloudo http://gigaom.com/2008/07/01/10-reasons-enterprises-arent-ready-to-trust-the-cloud/

• The Information Factories, by George Gildero http://www.wired.com/wired/archive/14.10/cloudware_pr.htmlo "When the network becomes as fast as the processor, the computer hollows out and spreads across the network."

- in 1993, from Sun Microsystems' CTO, Eric Schmidt (now CEO of Google)

• Above the Clouds: A Berkeley View of Cloud Computingo http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.html

Articles & Papers

• Google App Engine: http://appengine.google.com

• Videos from conference Google held in 2008o http://sites.google.com/site/io/

• Chat with developers from the App Engine teamo first Wednesday of the month: 3am next day (Dublin/London time).o third Wednesday of the month: 5pm (Dublin/London time).o IRC channel: #appengine on irc.freenode.neto http://en.wikipedia.org/wiki/List_of_IRC_clients

• Google Groupo http://groups.google.com/group/google-appengine/

• Paper on Bigtable: o http://labs.google.com/papers/bigtable.html

Resources for Google App Engine

Book (I)‏

Developing with Google App Engine

 

by Eugene Ciurana

Published:

Feb 2009

 

http://apress.com/

Book (II)‏

Google App Engine in Action

 

by Noah Gift & Michael Orr

To be published:

May 2009

 

http://www.manning.com/gift/

Book (III)‏

Google App Engine

 

by Dan Sanderson

To be published:

Sept 2009

 

http://oreilly.com/

2/27/2009

Wrapping upCastles in the Cloud

Credits• Photos used under Creative Commons license:

• Photo Creditso http://www.flickr.com/photos/notsogoodphotography/o http://www.flickr.com/photos/barto/o http://www.flickr.com/photos/kitoy/o http://www.flickr.com/photos/wolfgangstaudt/o http://www.flickr.com/photos/motumboe/o http://www.flickr.com/photos/ragnar1984/o http://www.flickr.com/photos/janekm/o http://www.flickr.com/photos/raylopez/o http://www.flickr.com/photos/pagedooley/o http://www.flickr.com/photos/gustty/o http://www.flickr.com/photos/mike9alive/o http://www.flickr.com/photos/tambako/o http://www.flickr.com/photos/nateo http://www.flickr.com/photos/freewine/

Cloudware is the arsenal of freedom

• Go guerillao Experiment with cloudware o Consider using cloudware for your new projectso Test-drive your favourite language on the App Engine SDKo Await availability for access to the production Java runtime

Contact Information

Kevin Noonan

kevin_noonan on Twitter

http://www.calbane.com