Post on 12-May-2015
description
Google Confidential and Proprietary
Building Scalable Apps withGoogle App Engine
David ChandlerDeveloper Programs Engineerdrfibonacci@google.com
CloudDevDay DetroitJun 11, 2011
1
1Monday, June 13, 2011
Google Confidential and Proprietary
Agenda
A word about the cloud
A Brief History of App Engine
Building with App Engine
What’s New in App Engine
Q&A
2
2Monday, June 13, 2011
3
3Monday, June 13, 2011
What iscloud
computing?
Place
Postage H
ere
IaaSPaaSSaaS
4
Infrastructure…
Platform…
Software…
… as a Service
4Monday, June 13, 2011
Google Confidential and Proprietary
App Engine is part of Google’s cloud platform
Source: Gartner AADI Summit Dec 2009 5
IaaS
PaaS
SaaS
5Monday, June 13, 2011
Google AppsThird party Apps: Google Apps Marketplace________
Google App Engine
Google Storage Prediction APIBigQuery
Google Confidential and Proprietary
Your Apps
IaaS
PaaS
SaaS
Google's Cloud Offerings
6
6Monday, June 13, 2011
Google Confidential and Proprietary
Not all "Platforms" are alike
7Monday, June 13, 2011
Google Confidential and Proprietary 8
Cloud development in a box
Hardware
Networking
Operating system
Application runtime• Java, Python
Static file serving
Services
Fault tolerance
Load balancing
8Monday, June 13, 2011
Google Confidential and Proprietary
What is Google App Engine?
Business: Google’s cloud computing platform
Technical: giant Web farm in front of world’s largest HashMap
Java developer: easiest and most scalable Java hosting environment
Startup developer: FREE to get started
App Engine expresses how Google thinks about infrastructure
9
9Monday, June 13, 2011
Google Confidential and Proprietary
Let’s do it
Sign up for AppEngine beta
Install Google Plugin for Eclipse
Create new Google Web application
Deploy it
10
10Monday, June 13, 2011
Google Confidential and Proprietary
What did we actually do?
Hot deployed an exploded WAR file
NO server setup
NO instance to provision
NO networking
NO J2EE config
NO start/stop
11
11Monday, June 13, 2011
Google Confidential and Proprietary
What makes AppEngine special?
12
12Monday, June 13, 2011
Google Confidential and Proprietary
BigTable: a Distributed Storage System
13
Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber
AbstractBigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this paper we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable.
Appeared in:OSDI'06: Seventh Symposium on Operating System Design and Implementation,Seattle, WA, November, 2006.
http://labs.google.com/papers/bigtable.html
13Monday, June 13, 2011
Google Confidential and Proprietary
High Replication Datastore
Default when you create a new app
Actively replicated to multiple data centers
14
14Monday, June 13, 2011
Google Confidential and Proprietary
No servers to configure
15
15Monday, June 13, 2011
Google Confidential and Proprietary 16
Easy deployment
OR
16Monday, June 13, 2011
Google Confidential and Proprietary
Powerful admin console
17
17Monday, June 13, 2011
Google Confidential and Proprietary
Distributed frontends
Edge Caching
Priority routing
Geographically close connection endpoint
18
18Monday, June 13, 2011
Google Confidential and Proprietary 19
Free to get started
6.5 CPU hrs/day
1 GB storage
650K URL Fetch calls
2,000 recipients emailed
1 GB/day bandwidth
100K tasks
$0.72 backend usage (9h on small instance)
~5M pageviews / month
19Monday, June 13, 2011
A Brief History of App Engine
20Monday, June 13, 2011
App Engine Growth
App Engine LaunchPython
DatastoreMemcachelogs export
2008 2009 2010
JavaDB Import
cron
Batch write/readHttps
Status-Dashboard
Task QueuesXMPP
incoming email
MultitenancyInstance Console
Always Onhi-perf imag
10 min tasks
BlobstoreAppstats
cursorsMapper
2011
Hi-ReplicationDatastore
Backends!Channel API
Files APIRemote API
Prosp Search
21Monday, June 13, 2011
100,000+ Active Developers per Month
By the Numbers
22Monday, June 13, 2011
200,000+Active apps per week
By the Numbers
23Monday, June 13, 2011
1.5B+Pageviews per day
By the Numbers
24Monday, June 13, 2011
Official Royal Wedding on App Engine
On Wedding day, served: • Up to 32k requests per second• 37.7 million pageviews • 13.7 million visitors
http://goo.gl/F1SGc
25Monday, June 13, 2011
Google Confidential and Proprietary
26Monday, June 13, 2011
Google Confidential and Proprietary
Building with App Engine∑
27
27Monday, June 13, 2011
Google Confidential and Proprietary 28
Language runtimes
Duke, the Java mascotCopyright © Sun Microsystems Inc., all rights reserved.
28Monday, June 13, 2011
Google Confidential and Proprietary 29
Specialized services
BlobstoreImages
Mail XMPP Task Queue
Memcache Datastore URL Fetch
User Service
29Monday, June 13, 2011
Google Confidential and Proprietary
Services are simple
UserService svc = UserServiceFactory.getUserService();
User authenticatedUser = svc.getUser();
30
30Monday, June 13, 2011
Google Confidential and Proprietary
Datastore APIs
Native• DatastoreServiceFactory.getDatastoreService().get(Key key)
• DatastoreServiceFactory.getDatastoreService().put(Entity entity)
• But…entities are not POJOs
JDO, JPA, or third party• Objectify (http://code.google.com/p/objectify-appengine)
• Twig, SimpleDS, Slim3, others
Objectify example (using DAOBase)
• ofy().put(Person p);
• ofy().get(Person.class, 238749);
31
31Monday, June 13, 2011
Google Confidential and Proprietary
Objectify goodies
Light• 36k jar, no external dependencies
Fast• no bytecode enhancement on start
Simple• No PersistenceManager lifecycle means no DI required
• No attach / detach hassle
• No “find to delete”
• No lazy fetching, proxy classes...
@Cache (uses App Engine Memcache)
@Embedded (objects as list tuples)
32
32Monday, June 13, 2011
Google Confidential and Proprietary
How to learn Datastore
Understand consistency vs. concurrency tradeoff• Selective denormalization sometimes required
• Use the power of the platform, don’t try to make it what it isn’t (relational) Batch gets / puts List properties for fast, lightweight indexes Retrieve parent entity from child key
Watch previous Google I/O presentations• Brett Slatkin 2009 on list properties and million-user fanout
• Book: see GWT Books page at code.google.com
33
33Monday, June 13, 2011
Google Confidential and Proprietary 34
Secure Data Connector (SDC)
34Monday, June 13, 2011
Google Confidential and Proprietary
How to do background processing?
Your application can’t start threads• Causes problems for some frameworks
• See will-it-play-in-appengine wiki
• No, you can’t run WebSphere or JBoss on AppEngine
AppEngine provides Task Queue API• Servlet-based
• Allows throttling
Backends (new!)
35
35Monday, June 13, 2011
Google Confidential and Proprietary
Queue a Task
Define queue in WEB-INF/queue.xml<queue>
<name>mail</name>
<rate>8/m</rate>
</queue>
Create worker servlet that handles task
Queue the taskqueue.add(url("/mail").param("key", key)…)
36
36Monday, June 13, 2011
Google Confidential and Proprietary
Is there an easier way?
37
37Monday, June 13, 2011
Google Confidential and Proprietary
Is there an easier way?
Vince Bonfanti’s Deferred task servlet
Now part of App Engine API!• com.google.appengine.api.taskqueue.DeferredTask
Task throttling can help with quotas
38
38Monday, June 13, 2011
Google Confidential and Proprietary
What’s New In App Engine?
39
39Monday, June 13, 2011
Google Confidential and Proprietary
Namespaces API
What’s wrong with this query in a multi-tenant app?
DELETE FROM TransactionsWHERE AcctID = 2347272
DELETE FROM TransactionsWHERE CustID = 2394872 AND AcctID = 2347272
Oops.
40
40Monday, June 13, 2011
Google Confidential and Proprietary22
Namespaces API
App ServerApp
Server
public void doFilter(ServletRequest req, ServletResponse resp, FilterChain chain) throws IOException, ServletException { NamespaceManager.set(...); chain.doFilter(request, response);}
SQL
41Monday, June 13, 2011
Google Confidential and Proprietary
Memcache entriesMemcache entry
Namespace API - Memcache
<namespace><app id><namespace><app id> <key> <value>
<key> <value>
<key> <value>
<key> <value>
<key> <value>
<key> <value>
<key> <value>
<key> <value>
42
42Monday, June 13, 2011
Google Confidential and Proprietary
Namespaces - Datastore
<namespace>
<namespace>
<namespace>
Row data
Single property index
Composite Index
<app id>
<app id>
<app id><namespace>
<namespace>
<namespace><app id>
<app id>
<app id>
<kind> <key> <properties>
<kind> <prop> <key><value>
<kind> <prop1> <prop2> <key>
43
43Monday, June 13, 2011
Google Confidential and Proprietary
Channel API
Asynchronous Server ⟺ Client Communication• Channel-based
• Bi-directional
Server• Send messages via ChannelService object
• Receive messages in a web hook
Client• JavaScript library
• Receive server messages in a callback
Built on Gmail chat client (Google Talk)
44
44Monday, June 13, 2011
Google Confidential and Proprietary
Channel API demo
45
www.typrx.com http://code.google.com/p/gwt-gae-channel/
45Monday, June 13, 2011
Google Confidential and Proprietary
Channel API - Server Interface
/** * ChannelService allows you to manage two-way connections * with clients. */public interface ChannelService { /** * Creates a channel associated with the provided applicationKey */ String createChannel(String applicationKey);
/** * Sends a ChannelMessage to the client. */ void sendMessage(ChannelMessage message);
/** * Parse the incoming message in request. This method * should only be called within a channel webhook. */ ChannelMessage parseMessage(HttpServletRequest request); }
4646Monday, June 13, 2011
Google Confidential and Proprietary
Channel API - Client (GWT)
47
http://code.google.com/p/gwt-gae-channel/
47Monday, June 13, 2011
Google Confidential and Proprietary
Android Tooling in GPE
In GPE 2.4 beta• Debug As | Local App Engine Connected Android Application
• Launches Android emulator
• Allows apps in the emulator to call App Engine dev server on localhost
See App Engine + Android session from this year’s Google I/O
48
48Monday, June 13, 2011
Google Confidential and Proprietary
App Engine is Leaving Preview (Beta)!
Full Google Product - 3 year deprecation support
99.95% SLA for all paid users
Operational and Developer Support available
New business-friendly Terms of Service
Billing via invoice
Rounded out feature set including Frontend and Backend capabilities
New sustainable pricing model
49
49Monday, June 13, 2011
Google Confidential and Proprietary
SLAs are more than just 9s
SLA is about transparency
HRD does not need planned downtime
Uptime will be measured by 1000s of various probes from around the globe
Customers will be able to contribute to our metrics
Upcoming Monitoring API to automatically monitor your own applications
50
50Monday, June 13, 2011
Google Confidential and Proprietary
Frontends and Backends
• Instances (running unit for both Frontends and Backends)o Have set amount of CPU and Memoryo Run your code
• Frontendso Scale dynamically based on your traffic
The App Engine Scheduler determines when to spin up or down Frontend instances
Controls will be added to allow customers to optimize the Scheduler for particular usage
• Backends (Launched at I/O with 1.5.0!)o Run based on admin controlo Choose CPU and Memory sizeo Long-running (no 30-sec HTTP request limit)
51
51Monday, June 13, 2011
Google Confidential and Proprietary
New Pricing Model Overview
• Usage types:o Free - similar to todayo Paid - $9/app/mo (+ usage) including SLAo Premier - $500/account/mo (+ usage) including operational
support
• No more CPU hourso Charging for size and number of running Frontends/Backends
(incorporating CPU + memory)o APIs (including Datastore) charged based on operations
• Add-onso For example: SSL for custom domains will be available for
monthly fee
52
52Monday, June 13, 2011
Google Confidential and Proprietary 53
Thank you!
http://code.google.com
53Monday, June 13, 2011