Automating Business with Google APIs

David Hodge

Thursday, March 7, 13

Thursday, March 7, 13

Thursday, March 7, 13

Thursday, March 7, 13

Thursday, March 7, 13

We automate business processes using cloud computing and

machine learning.

Dave & Bediako

Thursday, March 7, 13

IAAS and PAAS

- Amazon, Google, Microsoft

- DevOps, NoOps, OOPs

- Beware single point of failure - read SLAs

Thursday, March 7, 13

Google Docs Drive

• Google Docs API deprecated

• Google Drive Polyglot API - .NET, Java, Dart, Go, GWT, Java, Javascript, Objective-C, PHP, Python, Ruby

• ReSTful API - create and manage documents

Thursday, March 7, 13

Google Spreadsheets API

• Spreadsheets, Worksheets, List Row, Cell

• Create, Delete, Share

• List Based Feeds vs Cell Based Feeds

• Updating multiple cells with Batch

Thursday, March 7, 13

Google Spreadsheets API

• Gotchas

• Queries - age > 25 and height < 175

Thursday, March 7, 13

Google Apps Scripts

• Javascript cloud scripting language

• Automate workflows

• Customize spreadsheets

• Schedule Tasks

• Save data using Spreadsheets, JDBC, ScriptDB

Thursday, March 7, 13

Google App Engine

Thursday, March 7, 13

Google App Engine

Backends

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

Blobstore

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints

Images

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints

Images Logs

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints

Images Logs Mail

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints

Images Logs Mail

Memcache

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints

Images Logs Mail

Memcache Multitenancy

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints

Images Logs Mail

Memcache Multitenancy OAuth

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints

Images Logs Mail

Memcache Multitenancy OAuth

Prospective Search

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints

Images Logs Mail

Memcache Multitenancy OAuth

Prospective Search

Search

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints

Images Logs Mail

Memcache Multitenancy OAuth

Prospective Search

Search

Task Queues

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints

Images Logs Mail

Memcache Multitenancy OAuth

Prospective Search

Search

Task Queues

Users

Thursday, March 7, 13

Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints

Images Logs Mail

Memcache Multitenancy OAuth

Prospective Search

Search

Task Queues

Users

XMPP

Thursday, March 7, 13

One does not simply

Automate using Google APIs

Thursday, March 7, 13

How we build softwareA Customerhas a nick namehas a full namehas a date of birthhas a social security number which is sensitivecan be viewed or created or changed by a customer service representative

An Orderbelongs to a Customerhas a descriptioncan be viewed or created or changed by a customer service representative

An Order Line Item is auditedbelongs to an Orderhas a descriptionhas a price which is a numbercan be viewed or created or changed by a customer service representative

Dictation + Airlift on Google App Engine

Thursday, March 7, 13

Current processes manual and paper based for recording line items for accounts receivable and accounts payable.

Thursday, March 7, 13

Customer Widgets Spreadsheets

Thursday, March 7, 13

Widget Maker Spreadsheets

Thursday, March 7, 13

Evolution One - Cron<?xml version="1.0" encoding="UTF-8"?><cronentries>   <cron>    <url>/accounting/receivable</url>    <description>Process widgets sold for accounting line items</description>    <schedule>every day 12:30</schedule>    <timezone>America/New_York</timezone>  </cron>  <cron>    <url>/accounting/payable</url>    <description>Process line items for widget making workers</description>    <schedule>every day 03:30</schedule>    <timezone>America/New_York</timezone>  </cron></cronentries>

Thursday, March 7, 13

Evolution Two - Backends

• Use Backends to process long lived requests - No Request Deadlines

• Higher CPU up to 4.8GHz and Memory limits - up to 1GB

• Also using Spreadsheet Batch

Thursday, March 7, 13

Evolution Two - Backends

• Dynamic vs Resident - Dynamic Backends become alive only on request and are turned off when idle

Thursday, March 7, 13

Evolution Two - Backends

<backends> <backend name="accounting">   <options>     <dynamic>true</dynamic>     <fail-fast>true</fail-fast>    <class>B8</class>   </options> </backend></backends>

backends.xml

<?xml version="1.0" encoding="UTF-8"?><cronentries>   <cron>    <url>/accounting/receivable</url>    <description>Process widgets sold for accounting line items</description>    <schedule>every 20 minutes</schedule>    <timezone>America/New_York</timezone>

<target>accounting</target>  </cron>  <cron>    <url>/accounting/payable</url>    <description>Process line items for widget making workers</description>    <schedule>every 40 minutes</schedule>    <timezone>America/New_York</timezone>

<target>accounting</target>  </cron></cronentries>

cron.xml

Thursday, March 7, 13

Task Queues work discretely on a unit of work

Push - processes tasks based on rate configuration, auto scales

Pull - allows for specific time and timeframe to run

Thursday, March 7, 13

Task Push Config<queue-entries>  <queue>    <name>fooqueue</name>    <rate>1/s</rate>

<bucket-size>40</bucket-size>    <max-concurrent-requests>1</max-concurrent-requests>    <retry-parameters>      <task-retry-limit>7</task-retry-limit>      <task-age-limit>2</task-age-limit>    </retry-parameters>  </queue>  <queue>    <name>barqueue</name>    <rate>1/s</rate>    <retry-parameters>      <min-backoff-seconds>10</min-backoff-seconds>      <max-backoff-seconds>200</max-backoff-seconds>      <max-doublings>0</max-doublings>    </retry-parameters>  </queue>  </queue-entries>

queue.xml

Thursday, March 7, 13

Task Queue Headers

• X-AppEngine-QueueName, the name of the queue (possibly default)• X-AppEngine-TaskName, the name of the task, or a system-generated

unique ID if no name was specified• X-AppEngine-TaskRetryCount, the number of times this task has

been retried; for the first attempt, this value is 0. This number includes attempts where the task failed due to a lack of available instances and never reached the execution phase.

• X-AppEngine-TaskExecutionCount, the number of times this task has previously failed during the execution phase. This number does not include failures due to a lack of available instances.

• X-AppEngine-TaskETA, the target execution time of the task, specified in microseconds since January 1st 1970.

Thursday, March 7, 13

Task Push Queues

import com.google.appengine.api.taskqueue.Queue;import com.google.appengine.api.taskqueue.QueueFactory;import static com.google.appengine.api.taskqueue.TaskOptions.Builder.*;

//Discrete job for processing single widget to write customer line item    Queue queue = QueueFactory.getDefaultQueue();    queue.add(withUrl("/widgetorderforcustomerjob").param("widgetKey", key));

//Discrete job for processing single widget to write vendor line item    queue.add(withUrl("/widgetorderforvendorjob").param("widgetKey", key));

Thursday, March 7, 13

Google DocsGoogle App Engine

Automation Achieved

Receivable Line Items Job

Payable Line Items Job

Google Spreadsheet

Line Items Per Client

Google Spreadsheet

Line Items Per Vendor

Invoices

Payments

Thursday, March 7, 13

Experiment

EvolveArchitecture

Thursday, March 7, 13

Support

• Google Plus Hangouts

• Google Forums

• Fellow Developers

Thursday, March 7, 13

Some More Tips

• Python: Bulk Loader Tool to download/upload data - good for any flavor of GAE

• Get involved - NoOps doesn’t mean you don’t have to know your technology stack

• https://github.com/LucidTechnics/google-apps-util

• https://code.google.com/p/googleappengine/wiki/WillItPlayInJava

Thursday, March 7, 13

“Using no way as a way, having no limitation as limitation.”

- Bruce Lee

Thursday, March 7, 13

Official White House Photo by Pete SouzaThursday, March 7, 13

David Hodge@davezen1

Thursday, March 7, 13