PED: Performance Enhancing Development [Presentation]

Copyright © SAS Inst itute Inc. A l l r ights reserved.

PED: Performance Enhancing DevelopmentHow to improve productivity, efficiency, and results


When people talk about Data Science, they often focus on algorithms


Machine Learning Algorithms

Ensembles

Decision Tree

Support Vector

Machines

Neural Network

Random Forest Gradient Boosting Machine

Clustering

Penalized Regression

Factorization Machines

Features

Feature Engineering

Deep Learning


Automated Hyperparameter Tuning


x1

x2

y = f(x1) + g(x2)

Standard Grid Search

Typical Approaches

x1

x2

Random Searchx1

x2

Latin Hypercube

= individual model train and assessment


Optimization-based Autotuning• Formal optimization methods can more intelligently search the hyperparameter space to

find a combination which minimizes generalization error

Latin hypercube sampling of

hyperparameter space

Genetic Algorithm

Train Model

Assess on validation setOR

K-fold cross validation

Crossover and mutation

Evaluation

Iteration N(Generation)

Population

• Max time?• Max # evaluations?• Max # iterations?

Stop?Optimal set of

hyperparameter values

(Best Model)


Collaboration


Analytics has Multiple Languages

http://www.burtchworks.com/2017/06/19/2017-sas-r-python-flash-survey-results/


Workers

Controller

proc print data = hmeq (obs = 10);

run;

df = s.CASTable(‘hmeq’)

df.head(10)

df <- defCasTable(s, ‘hmeq’)

head(df, 10)

[table.fetch]

table.name = “hmeq”

from = 1 to = 10

Translated Action

APIs


Deployment into Production


Movie Rating Prediction Model

• 2006 - Announced $1M prize to whoever improved the accuracy by 10%

• 2009 - Prize won by 7-person team

• Winning model never implemented

“We evaluated some of the new methods offline but the

additional accuracy gains that we measured did not seem to

justify the engineering effort needed to bring them into a

production environment”

http://techblog.netflix.com/2012/04/netflix-recommendations-beyond-5-stars.html


ANALYTICS GOVERNANCE

PROCESS FLOW

Register

Candidate

Models

Compare

models

Declare

Champion

Model

Validate

Model/

Freeze

Version

Deploy

Model

Execute

Scoring

Monitor

Model

Performance

Retrain

Model

New

Model

Retire

Model


Benefits of Automation and Standardization

• Maximize the productivity of your team

• Minimize employee churn

• Maximize model reuse

• Minimize model handoff time

• Minimize time to market• “Best” models are in production sooner

• Minimize time spend on auditability and compliance to meet regulatory requirements

sas.com


Jared DeanPrincipal Data Scientist, SAS

[email protected]@jaredldean

http://www.sas.com/

PED: Performance Enhancing Development [Presentation]

Data & Analytics

Transcript of PED: Performance Enhancing Development [Presentation]