1 © Cloudera, Inc. All rights reserved. Engines, Algorithms, and Data Models Josh Wills | Senior...

1© Cloudera, Inc. All rights reserved.

Engines, Algorithms, and Data ModelsJosh Wills | Senior Director of Data Science

From Dimensional Modeling to Machine Learning


My First Data Warehouse


My Current Data Warehouse


The Rise of the Data Scientist


Data Scientist Supply vs. Data Scientist Demand


Moneyball and Data Science


Choosing The Right Metrics


1. Analyzing “Unstructured” Data Sources


2. Building Machine Learning Models


3. Turn Static Reports Into Analytical Applications


Answering More Questions in Less Time


How To Answer QuestionsLike A Data Scientist


1. Read and deserialize input data.

2. Project/filter input records.

3. Shuffle: serialize it, send over the network, deserialize it.

4. Apply aggregation logic.

5. Serialize output data.

The Life of a Data Processing Job


Handling the Cost of Serialization


The Traditional RDBMS Approach


The Cost of The Traditional RDBMS Approach


Query Scheduling and Exploratory Data Analysis


The Spark Approach


The Cost of the Spark Approach


The MapReduce Approach


MapReduce In The Hands of a Data Scientist


Example: Hive Multi-Insert


Our Goal: Public Transit for Questions


Data Modeling for Data Science


Motivating Example: Spelling Correction


Event Series Analytics


A Simple Star Schema for Spell Correction


The Combinatorial Explosion


• What parameters does this model need…• during the analysis phase?• during deployment?

• Some Candidates• Lag time between events• Similarity of queries• What else?

Designing the Spell Correction Data Product


A Supernova Schema for Search


Spell Correction in SQL


Exhibit: http://github.com/jwills/exhibit


Querying Nested Types with Impala


• Core Metric: # Outputs/ # Jobs• Measure on both an individual and

aggregate level• Drive the marginal cost of asking one

additional question towards zero• Point business analysts at output

tables for interactive analysis with Impala• Self-serve BI frees up resources

(compute + data science time)

Trading Up: From Data Analyst to Data Scientist


Thanks!@josh_wills

1 © Cloudera, Inc. All rights reserved. Engines, Algorithms, and Data Models Josh Wills | Senior...

Documents

Transcript of 1 © Cloudera, Inc. All rights reserved. Engines, Algorithms, and Data Models Josh Wills | Senior...