Data Science in Future Tense !

GalvanizeU Launch! 2014-10-29 gulaunch.splashthat.com

!

Paco Nathan @pacoid

Whither Data Science?

Whither Data Science?

twitter.com/josh_wills/status/198093512149958656

Whither Data Science?

twitter.com/josh_wills/status/198093512149958656

FLAW

ED

issue: aristotelian perspectives in a non-linear world…

Whither Data Science?

circa 2008: a large ad-tech firm, running one of the largest Hadoop instances in the cloud, execs said “Don’t bother” to dig into analysis of geo, clustering, time series, etc.

Whither Data Science?

circa 2008: a large ad-tech firm, running one of the largest Hadoop instances in the cloud, execs said “Don’t bother” to dig into analysis of geo, clustering, time series, etc. !We did anyway.

Whither Data Science?

circa 2008: a large ad-tech firm, running one of the largest Hadoop instances in the cloud, execs said “Don’t bother” to dig into analysis of geo, clustering, time series, etc. !We did anyway:

• people in SF don’t click online travel ads much, however, people in Dodge City do… a lot!

• largest customer segment: flag poles, portable generators, hammocks, sea salt, mail-order steaks, defibrillators

FLAW

ED

Whither Data Science?

primary sources for the notion:

Cleveland, W. S., “Data Science: an Action Plan for Expanding the Technical Areas of the Field of Statistics,” International Statistical Review (2001), 69, 21-26. http://cm.bell-labs.com/stat/doc/datascience.ps

Breiman L., “Statistical modeling: the two cultures”, Statistical Science (2001), 16:199-231. http://projecteuclid.org/euclid.ss/1009213726

…also good to mention John Tukey

Whither Data Science?

we have a long, long way yet to go:

So many problems that we encounter in industry can be represented as graphs… !Tensors provide means for representing multiple-edge graphs, ostensibly solving for a general case… !Even so, how much time have you spent working with tensors for data science apps?

wikipedia.org

Historical Arc 1: The Alchemists…

“Who has the crystal ball?”

Arc 1: Who has the crystal ball?

TL;DR: Nods to some people who envisaged and modeled our shared future…

Theory, Eight Decades Ago: what can be computed?

Haskell Curry haskell.org

Alonso Churchwikipedia.org

John Backusacm.org

David Turnerwikipedia.org

Praxis, Four Decades Ago: algebra for applicative systems

Pattie MaesMIT Media Lab

Reality, Two Decades Ago: web apps, ML, machine data

Arc 1: Who has the crystal ball?

spark.apache.org

A Brief History: Functional Programming for Big Data

databricks.com/blog/2014/10/10/spark-petabyte-sort.html

A Brief History: Smashing The Previous Petabyte Sort Record

spark.apache.org

Historical Arc 2: An Oblivoir Of Origins…

“Why are we here?”

Arc 2: Why are we here?

TL;DR: We share the delightful role of… !!speaking truth to power

Arc 2: Why are we here?

Reason 1: early 19th c. Prussian/Napoleonic “General Staff” organization => corporate IT silos !translated: too many people saying “That is not my concern.” !action: interdisciplinary teams tear down silos, surfacing insights

Arc 2: Why are we here?

Reason 2: 19th-20th c. statistics emphasized defensibility in lieu of predictability !translated: defend one’s job, not boost top-line revenue !action: focus on predictability; if you need to defend your job, you should be working elsewhere

Arc 2: Why are we here?

Reason 3: machine learning derives from several disciplines, but ultimately is a subset of optimization !translated: they couldn’t talk to each other very much, we have difficulty understanding them collectively !action: learn to leverage optimization theory, thoroughly

Arc 2: Why are we here?

Reason 4: university math curricula are still tilted toward Cold War priorities !translated: 2-3 years calculus weeds out the better mechanical engineering candidates who can build the most cost-effective ICBMs !action: leadership must embrace how to leverage advanced math for business use cases

Arc 2: Why are we here?

Reason 5: brogrammers tend to emphasize logical reasoning over analytic reasoning !translated: left-brained lopsidedness wins temporarily, then fails spectacularly !action: ask security to walk the brogrammers back to their cave

Arc 2: Why are we here?

Reason 6: people can make intuitive decisions in ~4 dimensions at most, period !translated: product managers as Steve Jobs wannabes are poisonous !action: leverage data science, visualization, machine learning with distributed systems at scale to address the high dimensionality of data

Arc 2: Why are we here?

Reason 7: embracing perpetual learning curves represents a promethean challenge !translated: learning is hard, and many organizations go to great lengths to minimize it !action: learn efficiently, continually, with a great thirst

Historical Arc 3: Be There Then…

“What happens next?”

Arc 4: What happens next?

TL;DR: Brace yourselves…

Arc 4: What happens next?

• Full stack… no, really

• You’ll work with functional programming and cloud-based notebooks

• Shift from modeling based on variance (batch) towards probabilistic approximation

• Early data scientists displace the old-school product managers

• IoT, drones, microsats: several orders of magnitude more data up ahead

• leave SF – the more interesting data science work to be accomplished is not here

Arc 4: What happens next?

Full stack… no, really

from visualization to virtualization, all points in-between

source: Microsoft

Arc 4: What happens next?

• Full stack… no, really

• You’ll work with functional programming and cloud-based notebooks

• Shift from modeling based on variance (batch) towards probabilistic approximation

• Early data scientists displace the old-school product managers

• IoT, drones, microsats: several orders of magnitude more data up ahead

• leave SF – the more interesting data science work to be accomplished is not here

Arc 4: What happens next?

You’ll work with functional programming and cloud-based notebooks

http://databricks.com/product

Arc 4: What happens next?

• Full stack… no, really

• You’ll work with functional programming and cloud-based notebooks

• Shift from modeling based on variance (batch) towards probabilistic approximation

• Early data scientists displace the old-school product managers

• IoT, drones, microsats: several orders of magnitude more data up ahead

• leave SF – the more interesting data science work to be accomplished is not here

Arc 4: What happens next?

Shift from modeling based on variance (batch) towards probabilistic approximation

highlyscalable.wordpress.com/2012/05/01/probabilistic-structures-web-analytics-data-mining/

Arc 4: What happens next?

• Full stack… no, really

• You’ll work with functional programming and cloud-based notebooks

• Shift from modeling based on variance (batch) towards probabilistic approximation

• Early data scientists displace the old-school product managers

• IoT, drones, microsats: several orders of magnitude more data up ahead

• leave SF – the more interesting data science work to be accomplished is not here

Arc 4: What happens next?

Early data scientists displace the old-school product managers

Arc 4: What happens next?

• Full stack… no, really

• You’ll work with functional programming and cloud-based notebooks

• Shift from modeling based on variance (batch) towards probabilistic approximation

• Early data scientists displace the old-school product managers

• IoT, drones, microsats: several orders of magnitude more data up ahead

• leave SF – the more interesting data science work to be accomplished is not here

Arc 4: What happens next?

IoT, drones, microsats: several orders of magnitude more data up ahead

airshipse.g., JP Aerospace, 40 km

atmostatse.g., Titan Aerospace, 20 km

microsatse.g., Planet Labs, 400 km

robotse.g., Blue River, 1 m

sensorse.g., Hortau, -0.3 m

dronese.g., HoneyComb, 120 m

Layered Sensing Networks

Arc 4: What happens next?

• Full stack… no, really

• You’ll work with functional programming and cloud-based notebooks

• Shift from modeling based on variance (batch) towards probabilistic approximation

• Early data scientists displace the old-school product managers

• IoT, drones, microsats: several orders of magnitude more data up ahead

• leave SF – the more interesting data science work to be accomplished is not here

Arc 4: What happens next?

leave SF – the more interesting data science work to be accomplished is not here

Summary?

After we’ve cleaned up data, formulated workflows in terms of monoids, used graph representation, and parallelized with a wealth of linear algebra, much of the heavy-lifting that remains on the clusters is in optimization

For example, deep learning @Google uses many layers of neural nets trained with gradient descent optimization Taming Latency Variability and Scaling Deep Learning Jeff Dean @Google (2013) youtu.be/S9twUcX1Zp0

Vector Quantization:

One advantage of quantum algorithms is to run large gradient descent problems in constant time… Reworking high-ROI apps to leverage lots of ML and large clusters, then SGD represents the datacenter cost basis, notably that part that scales…

Want to slash costs exponentially? Plug in quantum for a game-changer, maybe

Fast quantum algorithm for numerical gradient estimation Stephen P. Jordan Phys. Rev. Lett. 95, 050501 (2005) arxiv.org/abs/quant-ph/0405146 dwavesys.com

Vector Quantization:

Proposal: let’s drop clusters of quantum devices into lunar polar craters, so we can handle massive vector quantization workloads

• micro-kelvin environs

• near perpetual sunlight for energy sources

• park routers at L4

• approx. $15B to finance, i.e., ~6 days DoD budget

Vector Quantization:

We’ll just put this here… a couple o’ Googly projects in progress:

qCraft: Quantum Physics In Minecraft plus.google.com/u/1/+QuantumAILab/posts/grMbaaDGChH

Vector Quantization:

“We’re going back to the Moon. For good.”lunar.xprize.org

Resources

• spark.apache.org/community.html

• databricks.com/spark-training

• oreilly.com/go/sparkcert

Apache Spark community:

events:Strata EUBarcelona, Nov 19-21 strataconf.com/strataeu2014 Data Day Texas Austin, Jan 10 datadaytexas.com Strata CA San Jose, Feb 18-20 strataconf.com/strata2015 Spark Summit East NYC, Mar 18-19 spark-summit.org/east

Spark Summit 2015 SF, Jun 15-17 spark-summit.org

presenter:

Just Enough Math O’Reilly, 2014

justenoughmath.compreview: youtu.be/TQ58cWgdCpA

monthly newsletter for updates, events, conf summaries, etc.: liber118.com/pxn/

Enterprise Data Workflows with Cascading O’Reilly, 2013

shop.oreilly.com/product/0636920028536.do