Production and Beyond: Deploying and Managing Machine Learning Models
16
What happens after (initial) deployment
-
Upload
turi-inc -
Category
Technology
-
view
456 -
download
0
Transcript of Production and Beyond: Deploying and Managing Machine Learning Models
What happens after (initial) deployment
ML production life cycle
Evaluation
Monitoring
Deployment
Management
After deployment
Evaluate and track metrics over time.
React to feedback from deployed models.
Monitoring Management Evaluation
ML in production - 101Model
Historical Data
Predictions
LiveData
Feedback
Batch training
Real-time predictions
ML in production - 101Model
Historical Data
Real-time predictions
Batch training
PredictionsModel 2
LiveData
Key questions• When to update a model?• How to choose between existing models?• Answer: continuous evaluation and testing
What is evaluation?
Predictions Metric
+ Evaluation
What data?Which metric?
Evaluating a recommenderModel
Historical Data
Predictions
LiveData
Ranking loss
User engagement
Evaluating a recommenderModel
Historical Data
Predictions
LiveData
Ranking loss
User engagementOffline evaluation:
When to update modelOnline evaluation:Choosing between models
Updating ML modelsWhy update?• Trends and user tastes change over time• Model performance drops
When to update?• Track statistics of data over time• Monitor both offline & online metrics on live data• Update when offline metric diverges from online metrics
Choosing between ML models
Model 2
Model 1
2000 visits10% CTR
Group A
Everybody gets Model 2
2000 visits30% CTR
Group B
Strategy 1: A/B testing—select the best model and use it all the time
Choosing between ML models
A statistician walks into a casino…
Pay-off $1:$1000 Pay-off $1:$200 Pay-off $1:$500Play this 85% of
the timePlay this 10% of
the timePlay this 5% of the
time
Multi-armed bandits
Choosing between ML models
A statistician walks into an ML production environment
Pay-off $1:$1000 Pay-off $1:$200 Pay-off $1:$500
Use this 85% of the time
(Exploitation)
Use this 10% of the time
(Exploration)
Use this 5% of the time
(Exploration)
Model 1
Model 2
Model 3
MAB vs. A/B testingWhy MAB?• Continuous optimization, “set and forget”• Maximize overall reward
Why A/B test?• Simple to understand• Single winner• Tricky to do right
Other production considerations• Versioning• Logging• Provenance• Dashboards• Reports
“Machine learning: The high interest rate credit card of technical debt,” D. Sculley et al, Google, 2014“Two big challenges in machine learning,” Leon Bottou, ICML 2015 invited talk
Conclusions
Evaluation
Monitoring
Deploymen
t
Management
Dato Distributed&
Dato Predictive Services
A/B testing,multi-armed bandits
& much more
Dato – one stop shop for all stages of the ML life cycleSimple, platform agnostic interface
@datoinc, #DataSmt
