Using Task Queues and D3.js to build an analytics product on App Engine
-
Upload
warren -
Category
Technology
-
view
1.943 -
download
1
description
Transcript of Using Task Queues and D3.js to build an analytics product on App Engine
Using Task Queues and D3.js to build an
analytics product on App Engine
Warren EdwardsFounder, Waizee
TODAYHow Do We Handle All The Data?!?
Look at the Product
Why App Engine? Why D3.js?
Task Queues in App Engine
Code Samples
A Quiz!
Wrap Up
All of the data stored in the world currently about 4 zettabytes according to EMC.
At current growth rates, that data will surpass a yottabyte in 2030.
If each bit was represented by a grain of sand, a yottabyte would be about five percent the mass of the Moon.
Data surpasses human readability.
Software analyzes the data so humans
can focus on [ interpretation,
judgement, action ];
[ Show Product Demo ]
AWS AE Others
Team Experience
Automated scaling / spin-up
Automated load-balancing
Automated security
Task queue functionality
Why App Engine?
Honorable mention: AppScale path from AE
Google App Engine builds on Big Table
Why D3.js?Canvas Directly D3 PolyChart Math
plotlibSuper-
conductor
Team experience ( NONE )
Development effort
Maturity of library
Developer traction
Honorable mention: xCharts builds on D3.js with prepackaged chartsPHOTO CREDIT: http://www.flickr.com/photos/cusegoyle/
HTTP Request Task Queue
Full access to data store? YES YES
Atomic transactions? YES YES
Write in Python / Java / Go? YES YES
Maximum request lifetime 30-60 sec Up to 10 min
Timing of execution Immediate Up to 30 days
Retry Can be done manually, no policy
Automatic by policy
Concurrent requests No set limit Limited by policy
TQ build on App Engine's HTTP Requestbut liberate server use from user interaction
Introducing Task Queues (TQ)
Appeal of Task Queues
Well suited to number crunching● Long-lived jobs for running analytics● Full access to data store and messaging to build
on your existing App Engine knowledge● Save state back to Data Store and call the next
task
Task Queues allow you to crunch data "while you wait"
USERUPLOADS
DATATask#1
Task#2
. . . Task#N
RESULT
Save to Data Store
Pass Parameters
Cascade Tasks in Queue for Multipass Processing
Save to Data Store
Program Flow
Data Flow
DATA
STORE
Task#1
Task#2
. . .
Task#N
USERUPLOADS
DATARESULT
Process Tasks in Parallel Queues
Program Flow
Data Flow
Let's Look at a Code Sample
taskqueue.add(queue_name='analyze', url='/work1', params={'key': keyID})
class WorkerTheFirst(webapp2.RequestHandler): def post(self): keyID = self.request.get('key')
app = webapp2.WSGIApplication([('/', StartPage), ('/work1', WorkerTheFirst), ('.*', ErrPage) ])
Add Task into Queue
Define Task with Retrieval of Parameters
Associate Task with Handler
One Task Calls Anotherclass WorkerTheFirst(webapp2.RequestHandler): def post(self): keyID = self.request.get('key') # # First pass of work here # taskqueue.add(queue_name='analyze', url='/work2', params={'key': keyID})
class WorkerTheSecond(webapp2.RequestHandler): def post(self): keyID = self.request.get('key') # # Second pass of work here #
app = webapp2.WSGIApplication([('/', StartPage), ('/work1', WorkerTheFirst), ('/work2', WorkerTheSecond), ('.*', ErrPage) ])
total_storage_limit: 120G # Max for free apps is 500M
queue:# Queue for analyzing the incoming data- name: analyze rate: 35/s retry_parameters: task_retry_limit: 5 task_age_limit: 2h
# Queue for user behavior heuristics- name: heuristic rate: 5/s # Queue for doing maintenance on the data store or site- name: kickoff rate: 5/s
Setting up Task Queues in queue.yaml
svg = d3.select('body') .append('svg') .attr('class', 'circles') .attr('width', width) .attr('height', height)
svg.append('g').selectAll('circle') .data(data) .enter() .append('circle') .attr('transform', 'translate(' + pan + ', 0)') # pan allows moving te whole graph slightly # to account for long text labels
svg.selectAll('circle') .attr('cx', function(d) {return x(d.v2)}) .attr('cy', function(d) {return y(d.v1)}) .attr('r', dot_out) # dot_out scales the size .attr('fill', function(d) {return d.v4})
Core of the D3 Code
svg = d3.select('body') .append('svg') .attr('class', 'circles') .attr('width', width) .attr('height', height)
svg.append('g').selectAll('circle') .data(data) .enter() .append('circle') .attr('transform', 'translate(' + pan + ', 0)') # pan allows moving te whole graph slightly # to account for long text labels
svg.selectAll('circle') .attr('cx', function(d) {return x(d.v2)}) .attr('cy', function(d) {return y(d.v1)}) .attr('r', dot_out) # dot_out scales the size .attr('fill', function(d) {return d.v4})
Program Writes Its Own Code !
Axes are set heuristically by server software
Titles Are Set By Heuristic Text Analysis
var title = '{{ pagetitle }}'var subtitle = '{{ pagesubtitle }}'
var title = 'Google+ Rating More Important Metric Than Star Rating'var subtitle = 'Survey of Productivity Apps in the Chrome Web Store, Nov 2012'
In Django template
Rendered in Javascript to the browser
Labels were pulled Heuristically from input - not hard coded!
QUIZ !
Choose the Correct Task Queue Call
taskqueue.add(queue_name='analyze', url='/work2', param={key: keyID})
queue.add(queue='analyze', url='/work2', params={'key': keyID})
taskqueue.add(queue='analyze', url='/work2', param={'key': keyID})
taskqueue.add(queue_name='analyze', url='/work2', params={'key': keyID})
taskqueue.add(queue='analyze', url='/work2', params={key: keyID})
A
B
C
D
E
ANSWER IS ...
Choose the Correct Task Queue Call
taskqueue.add(queue_name='analyze', url='/work2', param={key: keyID})
queue.add(queue='analyze', url='/work2', params={'key': keyID})
taskqueue.add(queue='analyze', url='/work2', param={'key': keyID})
taskqueue.add(queue_name='analyze', url='/work2', params={'key': keyID})
taskqueue.add(queue='analyze', url='/work2', params={key: keyID})
A
B
C
D
E
Correct Answer is D
TQ Open a World of Possibilities
You can send tasks to different versions of your app → Automated test of new version of app before Go Live
You can access the queue’s usage data→ Your app can monitor its own consumption of tasks through QueueStatistics class
Task Queue + Crowdsource = ???→ Software application instructing humans !
Task Queues allow "while you wait" processing● Allow server task to run autonomously● Cascade tasks for multistep processing● Flexible functionality to create great products
Task Queues provide a great tool for automating the understanding of data
D3.js offers flexible, stable tool for viz of data● Works nicely with automated scripting● Lush visualizations but not pre-packaged● Leverage huge traction in San Francisco
D3.js is best platform for visualization using automated processing of data
We are your number cruncher in the cloud that understands your data and shows you only what is
most important.
Data SourcesThe Digital Universe in 2020: Big Data, Bigger Digital Shadows, and Biggest Growth in the Far East. IDC sponsored by EMC. December 2012
diameter and volume of the Moon: Wolfram Alpha
Yottabyte representation: Waizee calculations
2013 Gartner Magic Quadrant for Business Intelligence and Analytics Platforms
Photo Creditshttp://www.flickr.com/photos/pcoin/2066230078/sizes/m/in/photostream/
http://en.wikipedia.org/wiki/File:Visicalc.png
http://www.todayscampus.com/rememberthis/load.aspx?art=348
http://commons.wikimedia.org/wiki/File:Chocolate_chip_cookie.jpg
http://www.wpclipart.com/signs_symbol/checkmarks/checkmarks_3/checkmark_Bold_Brush_Green.png.html
http://www.flickr.com/photos/cusegoyle/