Google Cloud for Developers - Devfest Manila

142
Google Cloud for Developers Apps, Apps Marketplace, App Engine for Business, Storage, Prediction, Bigquery 2 Patrick Chanezon Developer Advocate [email protected] http://twitter.com/chanezon #devfest Manila July 6 2010 Tuesday, July 6, 2010

Transcript of Google Cloud for Developers - Devfest Manila

Page 1: Google Cloud for Developers - Devfest Manila

Google Cloud for Developers Apps, Apps Marketplace, App Engine for Business, Storage, Prediction, Bigquery

2

Patrick ChanezonDeveloper [email protected]://twitter.com/chanezon

#devfest ManilaJuly 6 2010

Tuesday, July 6, 2010

Page 2: Google Cloud for Developers - Devfest Manila

Tuesday, July 6, 2010

Page 3: Google Cloud for Developers - Devfest Manila

The benefits of Cloud Computing

Economics Pay for only what you use TCO OPEX vs CAPEXOperations Day to day: no maintenance Fighting fires: no PagersElasticityFocus on your Business

3

Tuesday, July 6, 2010

Page 4: Google Cloud for Developers - Devfest Manila

Enterprise Firewall

Enterprise Data Authentication Enterprise Services User Management

Google Apps Platform

Build and Buy all your enterprise cloud apps...

4

Tuesday, July 6, 2010

Page 5: Google Cloud for Developers - Devfest Manila

Enterprise Firewall

Enterprise Data Authentication Enterprise Services User Management

Google Apps Platform

Buy from Google

Google Apps for Business

Build and Buy all your enterprise cloud apps...

4

Tuesday, July 6, 2010

Page 6: Google Cloud for Developers - Devfest Manila

Buy from others

Google Apps Marketplace

Enterprise Firewall

Enterprise Data Authentication Enterprise Services User Management

Google Apps Platform

Buy from Google

Google Apps for Business

Build and Buy all your enterprise cloud apps...

4

Tuesday, July 6, 2010

Page 7: Google Cloud for Developers - Devfest Manila

Build your own

Google App Enginefor Business

Buy from others

Google Apps Marketplace

Enterprise Firewall

Enterprise Data Authentication Enterprise Services User Management

Google Apps Platform

Buy from Google

Google Apps for Business

Build and Buy all your enterprise cloud apps...

4

Tuesday, July 6, 2010

Page 8: Google Cloud for Developers - Devfest Manila

Customers want more Apps

5

GoogleApps

Businessin the cloud

Tuesday, July 6, 2010

Page 9: Google Cloud for Developers - Devfest Manila

Google Apps Marketplace

Tuesday, July 6, 2010

Page 10: Google Cloud for Developers - Devfest Manila

Marketplace Overview

Tuesday, July 6, 2010

Page 11: Google Cloud for Developers - Devfest Manila

Google Apps is Focused on Messaging & Collaboration

Gives every employee powerful messaging and collaboration tools without the usual IT hassle and cost

Tuesday, July 6, 2010

Page 12: Google Cloud for Developers - Devfest Manila

These customers want more great cloud apps...

Messaging

Collaboration

Accounting & Finance

Admin Tools

Calendaring / Meetings

Customer Management

Document Management

Productivity

Project Management

Sales & Marketing

Security & Compliance

Workflow

Tuesday, July 6, 2010

Page 13: Google Cloud for Developers - Devfest Manila

Steps to sell your app to Google Apps customers1. Build your app:- with any tools and hosting provider you want

2. Integrate your app:- add Single Sign On using OpenID (required)- access over a dozen integration points from Calendar, Contacts, Docs, etc. using OAuth (optional)

3. Sell your app:- to 2M+ businesses & 25M+ users- through Google Apps resellers- to wherever Google Apps goes- One-time fee of $100, 20% rev share starting 2H'10

Tuesday, July 6, 2010

Page 14: Google Cloud for Developers - Devfest Manila

Everything in the Cloud

Tuesday, July 6, 2010

Page 15: Google Cloud for Developers - Devfest Manila

Integrating with Google Apps & the Marketplace

Manifest

Tuesday, July 6, 2010

Page 16: Google Cloud for Developers - Devfest Manila

Demo Google Apps Marketplace

Tuesday, July 6, 2010

Page 17: Google Cloud for Developers - Devfest Manila

How it's done - Single Sign On

Tuesday, July 6, 2010

Page 18: Google Cloud for Developers - Devfest Manila

Single Sign OnOpenID with Google Apps

Welcome [email protected]

***********

Sounds complicated, but not hard in practice!

Tuesday, July 6, 2010

Page 19: Google Cloud for Developers - Devfest Manila

Single Sign OnOpenID Libraries

Language Libraries

Java OpenID4Java, Step2

.NET DotNetOpenAuth

PHP php-openid, php-openid-apps-discovery

Ruby ruby-openid,ruby-openid-apps-discovery

Any RPX, Ping Identity

Tuesday, July 6, 2010

Page 20: Google Cloud for Developers - Devfest Manila

Single Sign On - CodeStep 1 of 3 - Initialize libraryfunction getOpenIDConsumer() {  // Initialize client, storing associations  // in memcache  $store = new Auth_OpenID_MemcachedStore($MEMCACHE);  $consumer = new Auth_OpenID_Consumer($store);    // Enable Google Apps support   GApps_OpenID_EnableDiscovery($consumer);  return $consumer;}

Tuesday, July 6, 2010

Page 21: Google Cloud for Developers - Devfest Manila

Single Sign On - CodeStep 2 of 3 - Make the request// Create an auth request to the user's domain $auth_request = getOpenIDConsumer()->begin($domain);

// Request email address & name during login $ax = new Auth_OpenID_AX_FetchRequest; $attr = Auth_OpenID_AX_AttrInfo::make(      $AX_SCHEMA_EMAIL, 1, 1, 'email');$attr = Auth_OpenID_AX_AttrInfo::make(      $AX_SCHEMA_FIRSTNAME, 1, 1, 'first');$attr = Auth_OpenID_AX_AttrInfo::make(      $AX_SCHEMA_LASTNAME, 1, 1, 'last');$auth_request->addExtension($ax);

// Render Javascript/form to post request$form_html = $auth_request->htmlMarkup($REALM,      $RETURN_TO, false,       array('id' => 'openid_message')); print $form_html;

Tuesday, July 6, 2010

Page 22: Google Cloud for Developers - Devfest Manila

Single Sign On - CodeStep 3 of 3 - Handle response

// Parse the response from identity provider $response = getOpenIDConsumer()->complete($RETURN_TO); if ($response->status == Auth_OpenID_SUCCESS) {  // Extract data from response   $openid = $response->getDisplayIdentifier();  $ax = new Auth_OpenID_AX_FetchResponse();  $ax_resp = $ax->fromSuccessResponse($response);  $email = $ax_resp->data[$AX_SCHEMA_EMAIL][0];  $firstName = $ax_resp->data[$AX_SCHEMA_FIRSTNAME][0];  $lastName = $ax_resp->data[$AX_SCHEMA_LASTTNAME][0];  // Map to user in DB   $user = fetch_user($openid, $email, $firstName,      $lastName);} 

Tuesday, July 6, 2010

Page 23: Google Cloud for Developers - Devfest Manila

Single Sign On - CodeManifest

<ApplicationManifest>  <Name>Sassy Voice</Name>  <Description>Voice Mail &amp; Messaging</Description>  <Admin>    <Link rel="setup">http://voice.saasyapp.com/setup.php?domain=${DOMAIN_NAME}    </Link>  </Admin>    <Extension id="navlink" type="link">    <Name>SaasyVoice</Name>    <Url>http://voice.saasyapp.com/login.php?domain=${DOMAIN_NAME}    </Url>   </Extension>  <Extension id="realm" type="openIdRealm">    <Url>http://voice.saasyapp.com/</Url>  </Extension></ApplicationManifest>

Tuesday, July 6, 2010

Page 24: Google Cloud for Developers - Devfest Manila

Single Sign OnWhat we learned about users

Claimed ID: http://example.com/openid?id=12345

Email: [email protected]*First Name: BobLast Name: Dobbs

* Be sure to confirm or white-list trusted providers

Tuesday, July 6, 2010

Page 25: Google Cloud for Developers - Devfest Manila

How it's done - Data Access

Tuesday, July 6, 2010

Page 26: Google Cloud for Developers - Devfest Manila

2-Legged OAuth Delegating data access

<ApplicationManifest>  ...   <Scope id="userFeed">    <Url>https://apps-apis.google...    <Reason>To get a list of user...  </Scope>  <Scope id="contactsFeed">    <Url>https://www.google.com...     <Reason>To display names of...   </Scope>  <Scope id="docsFeed">     <Url>https://docs.google.com...     <Reason>To export a call log...   </Scope> </ApplicationManifest>

Tuesday, July 6, 2010

Page 27: Google Cloud for Developers - Devfest Manila

Data Access - CodeStep 1 of 2 - Initialize the client

function getOauthClient() {  $options = array(    'consumerKey' => $CONSUMER_KEY,    'consumerSecret' => $CONSUMER_SECRET,    'signatureMethod' => 'HMAC-SHA1',    'requestScheme' =>Zend_Oauth::REQUEST_SCHEME_HEADER,    'version' => '1.0');  $consumer = new Zend_Oauth_Consumer($options);  $token = new Zend_Oauth_Token_Access();  $httpClient = $token->getHttpClient($options);  return $httpClient;}

Tuesday, July 6, 2010

Page 28: Google Cloud for Developers - Devfest Manila

Data Access - CodeStep 2 of 2 - Fetching Users with Provisioning API// Initialize client$userClient = new Zend_Gdata_Gapps(getOauthClient());// Query feed for current user's domain$userQuery = new Zend_Gdata_Gapps_UserQuery(  getCurrentUserDomain());$usersFeed = $userClient->getUserFeed($userQuery);// Extract data from user feed$users = array();foreach ($usersFeed as $userEntry) {  $login = $userEntry->getLogin();  $name = $userEntry->getName();  $users[] = array(   'username' => $login->getUsername(),   'firstName' => $name->getGivenName(),   'lastName' => $name->getFamilyName(),   'admin' => $login->getAdmin());}

Tuesday, July 6, 2010

Page 29: Google Cloud for Developers - Devfest Manila

Integration Recap

With not a lot of code we added:

• Single Sign On with OpenID• Quicker setup with Attribute Exchange, Provisioning API• Integrated user data with Contacts API• Uploading spreadsheets with Docs API

Tuesday, July 6, 2010

Page 30: Google Cloud for Developers - Devfest Manila

Gadgets - your real estate in Google Apps

Tuesday, July 6, 2010

Page 31: Google Cloud for Developers - Devfest Manila

Gadgets

• Many types of gadgetsoGmail sidebaroCalendar sidebaroSites oSpreadsheets

• Two new types of gadgets available this week!oGmail contextual!oWave!

• All use the OpenSocial gadgets specification

Tuesday, July 6, 2010

Page 32: Google Cloud for Developers - Devfest Manila

Gmail contextual gadgets

• Detect e-mail content via regular expressions• Display and collect actionable business information

directly in Gmail, below each message

Tuesday, July 6, 2010

Page 33: Google Cloud for Developers - Devfest Manila

Gmail contextual gadgets

Tuesday, July 6, 2010

Page 34: Google Cloud for Developers - Devfest Manila

Gmail contextual gadgets

Tuesday, July 6, 2010

Page 35: Google Cloud for Developers - Devfest Manila

Calendar sidebar gadgets

• Can detect:o currently-displayed date range o currently-selected event

titleattendeesdate/time

Tuesday, July 6, 2010

Page 36: Google Cloud for Developers - Devfest Manila

Calendar sidebar gadgets

Tuesday, July 6, 2010

Page 37: Google Cloud for Developers - Devfest Manila

Calendar sidebar gadgets

Tuesday, July 6, 2010

Page 38: Google Cloud for Developers - Devfest Manila

Sell

Tuesday, July 6, 2010

Page 39: Google Cloud for Developers - Devfest Manila

Complete Manifest

<ApplicationManifest xmlns="http://schemas.google.com/ApplicationManifest/2009"><Name>SaaSy Voice</Name>  <Description>SaaSy Voice</Description>  <Support>    <Link rel="setup" href="https://.../setup.php?domain=${DOMAIN_NAME}"/>    <Link rel="manage" href="https://.../config.php?domain=${DOMAIN_NAME}"/>    <Link rel="deletion-policy" href="http://.../deletion-policy.php"/>    <Link rel="support" href="http://.../support.php"/>  </Support>

Tuesday, July 6, 2010

Page 40: Google Cloud for Developers - Devfest Manila

Complete Manifest(continued)

  <!-- Link in universal navigation -->   <Extension id="oneBarLink" type="link">     <Name>SaaSy Voice</Name>    <Url>http://.../index.php?domain=${DOMAIN_NAME}</Url>    <Scope ref="provisioningFeed"/>    <Scope ref="contactsFeed"/>     <Scope ref="docsFeed"/>  </Extension>   <!-- Declare our openid.realm -->   <Extension id="realm" type="openIdRealm">    <Url>http://voice.saasyapp.com/marketplace/</Url>  </Extension>

Tuesday, July 6, 2010

Page 41: Google Cloud for Developers - Devfest Manila

Complete Manifest(continued)

  <Scope id="provisioningFeed">    <Url>https://apps-apis.google.com/a/feeds/user/#readonly</Url>     <Reason>To get a list of users to provision accounts</Reason>  </Scope>  <Scope id="contactsFeed">     <Url>https://www.google.com/m8/feeds/</Url>    <Reason>To display names of people who called</Reason>  </Scope>  <Scope id="docsFeed">    <Url>https://docs.google.com/feeds/</Url>     <Reason>To export a call log</Reason>  </Scope></ApplicationManifest>

Tuesday, July 6, 2010

Page 42: Google Cloud for Developers - Devfest Manila

Listing in the Google Apps Marketplace

Tuesday, July 6, 2010

Page 43: Google Cloud for Developers - Devfest Manila

Listing in the Google Apps Marketplace

Tuesday, July 6, 2010

Page 44: Google Cloud for Developers - Devfest Manila

Reach 25 million Google Apps Users

"We've seen a very meaningful increase in high quality, non-paid lead flow directly attributable to Google Apps customers...Our customers cite Smartsheet's tight integration with Google's Data APIs as a key factor in their decision to purchase." Brent Frei, Smartsheet

Tuesday, July 6, 2010

Page 45: Google Cloud for Developers - Devfest Manila

Roadmap

Tuesday, July 6, 2010

Page 46: Google Cloud for Developers - Devfest Manila

Roadmap - General

• Recent LaunchesoGmail contextual gadgetsoOAuth support for Gmail's IMAP and SMTP

Tuesday, July 6, 2010

Page 47: Google Cloud for Developers - Devfest Manila

Roadmap - Billing

• GoalsoSimplified invoicing & payments for customersoConsistent experience across apps

• Planned 2nd-half of 2010oRequired for installable apps after transition periodoRevenue sharing starts once APIs adopted

• FeaturesoUnified billing for apps purchased in marketplaceoSupport for a variety of pricing & licensing modelsoAPIs for customization and up-selling

Tuesday, July 6, 2010

Page 48: Google Cloud for Developers - Devfest Manila

Learn More

Tuesday, July 6, 2010

Page 49: Google Cloud for Developers - Devfest Manila

Resources

• Business and Marketing o http://developer.googleapps.com/marketplace

• Technical and Codeo http://code.google.com/googleapps/marketplace

• Shopping!o http://www.google.com/appsmarketplace

• Don't have Google Apps?o http://www.google.com/a

Tuesday, July 6, 2010

Page 50: Google Cloud for Developers - Devfest Manila

Build your own

Google App Enginefor Business

Buy from others

Google Apps Marketplace

Enterprise Firewall

Enterprise Data Authentication Enterprise Services User Management

Google Apps Platform

Buy from Google

Google Apps for Business

Build and Buy all your enterprise cloud apps...

47

Tuesday, July 6, 2010

Page 51: Google Cloud for Developers - Devfest Manila

Google App Engine

Tuesday, July 6, 2010

Page 52: Google Cloud for Developers - Devfest Manila

Leveraging Google's Leadership in Cloud Computing• Massive data center operations• Purpose built hardware• Multi tenant software platform at Internet scale

49

Tuesday, July 6, 2010

Page 53: Google Cloud for Developers - Devfest Manila

Google App Engine

-Easy to build-Easy to maintain-Easy to scale

50

Tuesday, July 6, 2010

Page 54: Google Cloud for Developers - Devfest Manila

Cloud development in a box

51

• SDK & “The Cloud”• Hardware• Networking• Operating system• Application runtime

o Java, Python• Static file serving• Services• Fault tolerance• Load balancing

Tuesday, July 6, 2010

Page 55: Google Cloud for Developers - Devfest Manila

App Engine Services

BlobstoreImages

Mail XMPP Task Queue

Memcache Datastore URL Fetch

User Service

52

Tuesday, July 6, 2010

Page 56: Google Cloud for Developers - Devfest Manila

Always free to get started

~5M pageviews/month• 6.5 CPU hrs/day• 1 GB storage• 650K URL Fetch calls/day• 2,000 recipients emailed• 1 GB/day bandwidth• 100,000 tasks enqueued• 650K XMPP messages/day

53

Tuesday, July 6, 2010

Page 57: Google Cloud for Developers - Devfest Manila

Purchase additional resources *

* free monthly quota of ~5 million page views still in full effect54

Tuesday, July 6, 2010

Page 58: Google Cloud for Developers - Devfest Manila

0.5B+ daily Pageviews

250,000+ Developers

100,000+ Apps

By the numbers

55

Tuesday, July 6, 2010

Page 59: Google Cloud for Developers - Devfest Manila

App Engine

56

Tuesday, July 6, 2010

Page 60: Google Cloud for Developers - Devfest Manila

Socialwok

57

Tuesday, July 6, 2010

Page 61: Google Cloud for Developers - Devfest Manila

Social networking at scale

58

Tuesday, July 6, 2010

Page 62: Google Cloud for Developers - Devfest Manila

Social networking at scale

>62M Users

3.6M DAUs on Facebook1.9M DAUs on MySpace

Orkut, Bebo, Hi5,Friendster, Hyves, Ning, …

58

Tuesday, July 6, 2010

Page 63: Google Cloud for Developers - Devfest Manila

Chillingo CrystalGaming meets Social

Cogs

Guerilla Bob

Zombie Dash Angry Birds LITE Underground Meltdown

Mission Deep Sea Speed ForgeExtreme

Ravensword:The Fallen King

Angry Birds

59

Tuesday, July 6, 2010

Page 64: Google Cloud for Developers - Devfest Manila

gigy Socialize

60

Tuesday, July 6, 2010

Page 65: Google Cloud for Developers - Devfest Manila

gigy Socialize - traffic

61

Tuesday, July 6, 2010

Page 66: Google Cloud for Developers - Devfest Manila

gigy Socialize - traffic

61

Tuesday, July 6, 2010

Page 67: Google Cloud for Developers - Devfest Manila

App Engine

62

Tuesday, July 6, 2010

Page 68: Google Cloud for Developers - Devfest Manila

Google App Engine for Business

Tuesday, July 6, 2010

Page 69: Google Cloud for Developers - Devfest Manila

Build your Enterprise Apps on Google

• Easy to Build - Java standards

• Easy to Deploy - push-button deployment

• Easy to Scale - from small apps to millions of users

64

Tuesday, July 6, 2010

Page 70: Google Cloud for Developers - Devfest Manila

Google App Engine for Business

• Centralized administration - controls

• Reliability and support - SLA, Premium support

• Secure by default - only your users

• Pricing that makes sense - pay only for what you use

• Enterprise features - hosted SQL, SSL on your domain

65

Tuesday, July 6, 2010

Page 71: Google Cloud for Developers - Devfest Manila

Understanding the Cloud Computing Landscape

IaaS

PaaS

SaaS

Source: Gartner AADI Summit Dec 2009

66

Tuesday, July 6, 2010

Page 72: Google Cloud for Developers - Devfest Manila

Google Storage for DevsMachine Learning

BigQuery

Google App Engine

Google's Cloud Offerings

IaaS

PaaS

SaaS

1. Our Apps2. 3rd party Apps: Google Apps Marketplace3. ________

67

Tuesday, July 6, 2010

Page 73: Google Cloud for Developers - Devfest Manila

Google Storage for DevsMachine Learning

BigQuery

Google App Engine

Google's Cloud Offerings

IaaS

PaaS

SaaS

Your Apps

1. Our Apps2. 3rd party Apps: Google Apps Marketplace3. ________

67

Tuesday, July 6, 2010

Page 74: Google Cloud for Developers - Devfest Manila

Domain Console

68

Tuesday, July 6, 2010

Page 75: Google Cloud for Developers - Devfest Manila

Domain Console

Like the regular admin consoleDesigned to manage enterprises with a portfolio of apps• Keep track of all apps in a domain• Access Control: view apps, deploy• Global Settings: apply to all apps in the domain • Billing rolling up to single account• DNS configuration done only once: *.ext.example.com• All apps by default for logged in users from domain

69

Tuesday, July 6, 2010

Page 76: Google Cloud for Developers - Devfest Manila

Google Apps Integration• SSO/SSO delegation• APIs for most Google Apps for integration 

70

Tuesday, July 6, 2010

Page 77: Google Cloud for Developers - Devfest Manila

Federate your on-premise data

71

Tuesday, July 6, 2010

Page 78: Google Cloud for Developers - Devfest Manila

Secure Data Connector (SDC)

72

Tuesday, July 6, 2010

Page 79: Google Cloud for Developers - Devfest Manila

Using Secure Data Connector

73

Tuesday, July 6, 2010

Page 80: Google Cloud for Developers - Devfest Manila

Using Secure Data Connector

Installation- Determine access rules- Configure and install SDC

73

Tuesday, July 6, 2010

Page 81: Google Cloud for Developers - Devfest Manila

Using Secure Data Connector

Installation- Determine access rules- Configure and install SDC

Getting ready to serve- SDC opens SSL tunnel

73

Tuesday, July 6, 2010

Page 82: Google Cloud for Developers - Devfest Manila

Using Secure Data Connector

Installation- Determine access rules- Configure and install SDC

Getting ready to serve- SDC opens SSL tunnel

Serving- User request sent to App Engine- User authenticated- App makes request through tunnel- SDC performs access checks- Results returned

73

Tuesday, July 6, 2010

Page 83: Google Cloud for Developers - Devfest Manila

App Engine for Business Pricing

Intranet apps:Each app costs $8 / active user / monthCapped at $1,000 / month (i.e. users above 125 are free)Apps are auth-restricted to domain usersDevelopment is freeOverage charges on Background Analysis/Storage

Non intranet apps (external/public/ISV apps):Pricing TBD Postpaid (i.e. billed at the end of month)

74

Tuesday, July 6, 2010

Page 84: Google Cloud for Developers - Devfest Manila

App Engine for Business Support and SLAPaid Support Email based 1000$/month 1h response time on operational issues 8h on development issues SLA 99.9% uptime Service credits from 10% to 100% refund of monthly bill

75

Tuesday, July 6, 2010

Page 85: Google Cloud for Developers - Devfest Manila

Gadgets JS Maps APISearch KML 3DApp Engine

Google Developer Qualification

Chrome Extensions

76

Tuesday, July 6, 2010

Page 86: Google Cloud for Developers - Devfest Manila

Two years in review

77

Apr 2008 Python launchMay 2008 Memcache, Images APIJul 2008 Logs exportAug 2008 Batch write/deleteOct 2008 HTTPS supportDec 2008 Status dashboard, quota detailsFeb 2009 Billing, larger filesApr 2009 Java launch, DB import, cron support, SDCMay 2009 Key-only queriesJun 2009 Task queuesAug 2009 Kindless queriesSep 2009 XMPPOct 2009 Incoming emailDec 2009 BlobstoreFeb 2010 Datastore cursors, AppstatsMar 2010 Read policies, IPv6May 2010 App Engine for Business

Tuesday, July 6, 2010

Page 87: Google Cloud for Developers - Devfest Manila

An evolving platform

78

Tuesday, July 6, 2010

Page 88: Google Cloud for Developers - Devfest Manila

App Engine Roadmap

79

SSL for your domainBackground servers

Reserved instances

Control datastore availability vs. latency trade-offs

Mapping operations across datasetsDatastore dump and restore facility

Raise request/response size limits for some APIs

Improved monitoring/alerting

Channel APIBuilt-in support for OAuth & OpenID

Tuesday, July 6, 2010

Page 89: Google Cloud for Developers - Devfest Manila

Google Storage

Tuesday, July 6, 2010

Page 90: Google Cloud for Developers - Devfest Manila

Google Storage• Cloud-based binary object store

oStructured as buckets and objectsoMany buckets, many objects, large objects

• You control your dataoPrivate, shared, or publicoGet your data back out at any time

• For developersoRESTful APIoMany SDKs + toolso Integration with other Google services

Tuesday, July 6, 2010

Page 91: Google Cloud for Developers - Devfest Manila

Google Storage Benefits

High Performance and Scalability backed by Google infrastructure

Flexible Authentication & Sharing Models

Get Started Fast withGoogle & 3rd Party Utilities

Tuesday, July 6, 2010

Page 92: Google Cloud for Developers - Devfest Manila

Demo: Get Started In < 1 Minute• click emailed invitation link

• read & accept Terms Of Service

• start using GS Manager

Tuesday, July 6, 2010

Page 93: Google Cloud for Developers - Devfest Manila

Google services using Google Storage

Partner ReportingData Liberation Haiti Relief Imagery

Google BigQuery

Google Prediction API

Partner Reporting

Tuesday, July 6, 2010

Page 94: Google Cloud for Developers - Devfest Manila

Some current users

Tuesday, July 6, 2010

Page 95: Google Cloud for Developers - Devfest Manila

Technical Overview

Tuesday, July 6, 2010

Page 96: Google Cloud for Developers - Devfest Manila

Google Storage Overview

• Fast, scalable, highly available object storeoObjects of any type and practically any sizeo Lots of objects, lots of bucketsoAll data replicated to multiple US data centersoRead-your-writes data consistency

• Easy, flexible authentication and sharingoKey-based authenticationoAuthenticated downloads from a web browseroSharing with individuals and groups

• Google products and 3rd party tools/services oCompatible with many available tools and librariesoGetting started toolkit

Tuesday, July 6, 2010

Page 97: Google Cloud for Developers - Devfest Manila

API Concepts

• RESTful APIoVerbs: GET, PUT, POST, HEAD, DELETEoResources: identified by URI

• BucketsoFlat containers

• ObjectsoAny type, practically any size

• Access Control for Google AccountsoGoogle Groups

• Two Ways to Authenticate RequestsoSign request using access keysoWeb browser login

Tuesday, July 6, 2010

Page 98: Google Cloud for Developers - Devfest Manila

Sample Signed Request

PUT /mybucket/My/Long/Object/Name HTTP/1.1Host: commondatastorage.googleapis.com:443Accept-Encoding: identityDate: Sat, 08 May 2010 19:04:21 GMTContent-Length: 28Content-Type: text/plain Authorization:GOOG1GOOG4622809698762217:J+y3mj5GThfI6Ed1MqLi7JpCq5Y= This is my object's content.

Tuesday, July 6, 2010

Page 99: Google Cloud for Developers - Devfest Manila

Sharing and ACLs

• Data can be private or shared

• Bucket ACL determines:owho can list objects (READ) owho can create / delete objects (WRITE)owho can read / write bucket ACL (FULL_CONTROL)

• Object ACL determines:owho can read objects (READ)owho can read / write object ACL (FULL_CONTROL)

Tuesday, July 6, 2010

Page 100: Google Cloud for Developers - Devfest Manila

Read-Your-Writes Consistency

• Once a write succeeds, all future reads will see a snapshot that includes that write... no matter what replica they talk to

• Once any reader sees a result (even if the write previously appeared to fail) then all future readers will see a snapshot that includes the write

Tuesday, July 6, 2010

Page 101: Google Cloud for Developers - Devfest Manila

Interoperability Gives You Choice

• Data LiberationoYou shouldn't be locked in by your choice of

storage provider

• Choice of toolsoYou should be able to use the same tools to

manage your data, regardless of where you keep it

Tuesday, July 6, 2010

Page 102: Google Cloud for Developers - Devfest Manila

Demo: Using Google Storage

Tuesday, July 6, 2010

Page 103: Google Cloud for Developers - Devfest Manila

Computing in the Cloud: App Engine Sampleos.environ['BOTO_CONFIG'] = 'boto.cfg'from boto import storage_uri# <other imports omitted>

class MainPage(webapp.RequestHandler):  def get(self):    self.response.out.write('<html><body>')    uri = storage_uri('gs://pub/shakespeare/rose.txt')    poem = uri.get_contents_as_string()    self.response.out.write('<pre>' + poem + '</pre>')    self.response.out.write('</body></html>')

def main():  application = webapp.WSGIApplication([('/', MainPage)])  run_wsgi_app(application)

if __name__ == "__main__":  main()

Tuesday, July 6, 2010

Page 104: Google Cloud for Developers - Devfest Manila

Pricing and Availability

• Pay as you go pricing• Storage - $0.17/GB/month• Network

o Upload data to Google$0.10/GB

o Download data from Google$0.15/GB for Americas and EMEA$0.30/GB for APAC

• Requestso PUT, POST, LIST - $0.01 per 1,000 Requestso GET, HEAD - $0.01 per 10,000 Requests

• Free storage (up to 100GB) during preview periodo No SLA

• http://code.google.com/apis/storage

Tuesday, July 6, 2010

Page 105: Google Cloud for Developers - Devfest Manila

What's Coming Up

• Service Level Agreement

• Support

• Available to Premium Apps Customers

• Technical Features:oGroup support in ACLsoResumable uploadsoAdditional regions

Tuesday, July 6, 2010

Page 106: Google Cloud for Developers - Devfest Manila

Google BigQuery and Prediction APIs

Tuesday, July 6, 2010

Page 107: Google Cloud for Developers - Devfest Manila

Overview

• Big Data - Challenging and Important

• Google has tools for deep data analysis

• Now you can use these tools

Tuesday, July 6, 2010

Page 108: Google Cloud for Developers - Devfest Manila

Overview

• Big Data - Challenging and Important

• Google has tools for deep data analysis

• Now you can use these tools

• Announcing two new APIs to get more from your data: 1.BigQuery2.Prediction API

Tuesday, July 6, 2010

Page 109: Google Cloud for Developers - Devfest Manila

Benefits

• Built on Google technology

• Scalability

• Security

• Sharing

• Easy integration with Google App Engine, Google Spreadsheets, ....

Tuesday, July 6, 2010

Page 110: Google Cloud for Developers - Devfest Manila

Using Your Data with BigQuery & Prediction API

1. Upload

2. Process

Upload your datato Google Storage

Import to tablesTrain a model

Run queriesMake predictions3. Act

Your Data

BigQuery Prediction API

Your Apps

Tuesday, July 6, 2010

Page 111: Google Cloud for Developers - Devfest Manila

BigQueryInteractive Analysis of Big Data

Tuesday, July 6, 2010

Page 112: Google Cloud for Developers - Devfest Manila

Big Data is Challenging

Starts with Scale

Tuesday, July 6, 2010

Page 113: Google Cloud for Developers - Devfest Manila

Many Use Cases ...

Spam TrendsDetection

Web Dashboards

Network Optimization

Interactive Tools

Tuesday, July 6, 2010

Page 114: Google Cloud for Developers - Devfest Manila

Demo: Analyzing M-LabAn open platform for advanced network research http://www.measurementlab.net/

Tuesday, July 6, 2010

Page 115: Google Cloud for Developers - Devfest Manila

Demo: Exploring M-Lab

Tuesday, July 6, 2010

Page 116: Google Cloud for Developers - Devfest Manila

Key Capabilities of BigQuery

• Scalable: Billions of rows• Fast: Response in seconds

• Simple: Queries in SQL

• Web Serviceo RESTo JSON-RPCo Google App Scripts

Tuesday, July 6, 2010

Page 117: Google Cloud for Developers - Devfest Manila

Using Your Data with BigQuery

1. Upload

2. Import

Upload to Google Storage

Import data into a BigQuery Table- No need to define indices, keys, etc..

Execute queries via APIs- No provisioning machines or resources3. Query

Tuesday, July 6, 2010

Page 118: Google Cloud for Developers - Devfest Manila

Writing Queries

Compact subset of SQLo SELECT ... FROM ...

WHERE ... GROUP BY ... ORDER BY ...LIMIT ...;

Common functionsoMath, String, Time, ...

Statistical approximationsoTOPoCOUNT DISTINCT

Tuesday, July 6, 2010

Page 119: Google Cloud for Developers - Devfest Manila

API in a Minute

GET /bigquery/v1/tables/{table name}

GET /bigquery/v1/query?q={query}Sample JSON Reply:{  "results": {    "fields": { [      {"id":"COUNT(*)","type":"uint64"}, ... ]    },    "rows": [      {"f":[{"v":"2949"}, ...]},      {"f":[{"v":"5387"}, ...]}, ... ]  }}

Also supports JSON-RPC

Tuesday, July 6, 2010

Page 120: Google Cloud for Developers - Devfest Manila

Security and Privacy

Standard Google Authentication• Client Login• OAuth• AuthSub

HTTPS support• protects your credentials• protects your data

Use Google Storage for Developers to manage access

Tuesday, July 6, 2010

Page 121: Google Cloud for Developers - Devfest Manila

Large Corpus Analysis

Wikimedia Revision history data from: http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-meta-history.xml.7z

Wikimedia Revision History

Tuesday, July 6, 2010

Page 122: Google Cloud for Developers - Devfest Manila

Using BigQuery ShellPython DB API 2.0 + B. Clapper's sqlcmdhttp://www.clapper.org/software/python/sqlcmd/

Tuesday, July 6, 2010

Page 123: Google Cloud for Developers - Devfest Manila

BigQuery from a Spreadsheet

Tuesday, July 6, 2010

Page 124: Google Cloud for Developers - Devfest Manila

BigQuery from a Spreadsheet

Tuesday, July 6, 2010

Page 125: Google Cloud for Developers - Devfest Manila

BigQuery Recap

• Interactive analysis of very large data sets• Simple SQL query language

• APIs enable a variety of use cases

Tuesday, July 6, 2010

Page 126: Google Cloud for Developers - Devfest Manila

Google Prediction APIMachine learning as a web service

Tuesday, July 6, 2010

Page 127: Google Cloud for Developers - Devfest Manila

Prediction API 101

• Google's sophisticated machine learning algorithms

• Available as an on-demand RESTful HTTP web service

• Train a model offline/asynchronously

• Predict results in real-time

"french"Prediction API"Tous pour un, un pour tous, c'est notre devise."

Tuesday, July 6, 2010

Page 128: Google Cloud for Developers - Devfest Manila

How does it work?

"english" The quick brown fox jumped over the lazy dog.

"english" To err is human, but to really foul things up you need a computer.

"spanish" No hay mal que por bien no venga.

"spanish" La tercera es la vencida.

? To be or not to be, that is the question.

? La fe mueve montañas.

The Prediction APIfinds relevantfeatures in the sample data during training.

The Prediction APIlater searches forthose featuresduring prediction.

Tuesday, July 6, 2010

Page 129: Google Cloud for Developers - Devfest Manila

A virtually endless number of applications...

CustomerSentiment

TransactionRisk

SpeciesIdentification

MessageRouting

Legal DocketClassification

SuspiciousActivity

Work RosterAssignment

RecommendProducts

PoliticalBias

UpliftMarketing

EmailFiltering

Diagnostics

InappropriateContent

CareerCounselling

ChurnPrediction

... and many more ...

Tuesday, July 6, 2010

Page 130: Google Cloud for Developers - Devfest Manila

Three simple steps to use the Prediction API

1. Upload

2. Train

Upload your training data toGoogle Storage

Build a model from your data

Make new predictions

prediction/v1/train/{}POST : a training request

prediction/v1/query/{}GET : model infoPOST : a prediction request

Use the API, gsutil or any compatible utility to upload your data to Google Storage

3. Predict

Tuesday, July 6, 2010

Page 131: Google Cloud for Developers - Devfest Manila

Prediction API Demo Automatically categorize and respond to emails by language

• Customer: ACME Corp, a multinational organization• Goal: Respond to customer emails in their language• Data: Many emails, tagged with their languages

• Outcome: Predict language and respond accordingly

Tuesday, July 6, 2010

Page 132: Google Cloud for Developers - Devfest Manila

Step 1: UploadUpload your training data to Google Storage

• Training data: outputs and input features • Data format: comma separated value format (CSV)

$ head -n 2 ${data}"english","To err is human, but to really ...""spanish","No hay mal que por bien no venga."

Upload to Google Storage$ gsutil cp ${data} gs://io10/${data}

Tuesday, July 6, 2010

Page 133: Google Cloud for Developers - Devfest Manila

Step 2: TrainCreate a new model by training on data

To train a model:

POST prediction/v1/train/${data}

Training runs asynchronously. To see if it has finished:

GET prediction/v1/query/${data}

{"data": {    "resource": {      "data": "${data}",       "modelinfo": "estimated accuracy: ${acc}"}}}

Tuesday, July 6, 2010

Page 134: Google Cloud for Developers - Devfest Manila

Step 3: PredictApply the trained model to make predictions on new data

POST prediction/v1/query/${data}

{ data : {   "instance" : {    "input" : { "text" : [      "J'aime X! C'est le meilleur" ]}}}}

Tuesday, July 6, 2010

Page 135: Google Cloud for Developers - Devfest Manila

Step 3: PredictApply the trained model to make predictions on new data

POST prediction/v1/query/${data}

{ data : {   "instance" : {    "input" : { "text" : [      "J'aime X! C'est le meilleur" ]}    "output" :    {"output_label" : "french"}}}}

Tuesday, July 6, 2010

Page 136: Google Cloud for Developers - Devfest Manila

Step 3: PredictApply the trained model to make predictions on new data

import httplib

header = {"Content-Type" : "application/json"}#...put new data in JSON format in params variableconn = httplib.HTTPConnection("www.googleapis.com")conn.request("POST",  "/prediction/v1/query/${data}",  params, header)print conn.getresponse()

Tuesday, July 6, 2010

Page 137: Google Cloud for Developers - Devfest Manila

Prediction API Capabilities

Data• Input Features: numeric or unstructured text• Output: up to 100s of discrete categories

Training• Many machine learning techniques• Automatically selected • Performed asynchronously

Access from many platforms:• Web app from Google App Engine• Apps Script (e.g. from Google Spreadsheet)• Desktop app

Tuesday, July 6, 2010

Page 138: Google Cloud for Developers - Devfest Manila

Prediction API and BigQuery Demo: Tagger

Input Data: http://delic.io.us/chanezon–6000 urls, 14000 tags in 6 years

Analyze my delicious tags–use delicious API to get all tagged urls–cleanup data, resize (100Mb limit)–PUT data in Google storage–Define table–analyze

Predict how I would tag a technology article–input is tag,url,text–send new url and text–get predicted tag

Tuesday, July 6, 2010

Page 139: Google Cloud for Developers - Devfest Manila

Get the BigQuery & Prediction APIs

• Preview, opened to a limited number of developers• You need a Google Storage for Developers account• To request access and get more information, go to:

o http://code.google.com/apis/bigqueryo http://code.google.com/apis/prediction

Tuesday, July 6, 2010

Page 140: Google Cloud for Developers - Devfest Manila

Acknowledgement

Thanks to many Googlers from their slides, most available at the Google IO 2010 website

131

Tuesday, July 6, 2010

Page 141: Google Cloud for Developers - Devfest Manila

Thank you

Read morehttp://code.google.com/appengine/

Contact infoPatrick ChanezonDeveloper [email protected]://twitter.com/chanezon

Questions?

132

Tuesday, July 6, 2010

Page 142: Google Cloud for Developers - Devfest Manila

Tuesday, July 6, 2010