Serverless machine learning operations
-
Upload
stepan-pushkarev -
Category
Software
-
view
127 -
download
1
Transcript of Serverless machine learning operations
![Page 1: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/1.jpg)
Serverless Machine Learning Operations
by Stepan Pushkarev CTO of Hydrosphere.io
![Page 2: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/2.jpg)
Mission: Accelerate Machine Learning to Production
Opensource Products:- Mist: Spark Compute as a Service- ML Lambda: ML Function as a Service - Sonar: Data and ML Monitoring
Business Model: Subscription services and hands-on consulting
About
![Page 3: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/3.jpg)
Ops folks here?
Machine Learning nerds here?
VP/Managers/Strategy?
![Page 4: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/4.jpg)
Development Operations are well studied
![Page 5: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/5.jpg)
Machine Learning operations are ad hoc
● Research phase -> productization phase
● Scripts driven ./bin/spark-submit
python train.py
● Raw SQL / HiveQL / SQL on Hadoop
● Automated with Cron and/or Workflow
Managers
● Hosted Notebooks culture
![Page 6: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/6.jpg)
ML Project Time to Market
![Page 7: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/7.jpg)
ML Project Time to Market
![Page 8: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/8.jpg)
ML Project Time to Market
![Page 9: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/9.jpg)
- Go to production strategy from the Day 1
- Training: Serverless Spark Compute
- Serving/inferencing: Serverless ML Lambdas
Agenda
![Page 10: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/10.jpg)
Why does business hire data scientists?
![Page 11: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/11.jpg)
Why do companies hire data scientists?
To make products smarter.
![Page 12: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/12.jpg)
What is a deliverable of data scientist?
Academic paper
ML Model R/Python script
Jupiter Notebook
BI Dashboard
![Page 13: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/13.jpg)
How to move this to prod?
Academic paper?
ML Model? R/Python script?
Jupiter Notebook?
BI Dashboard?
![Page 14: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/14.jpg)
Tragedy 1: Engineer to re-implement R/Python script
![Page 15: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/15.jpg)
Tragedy 2: Notebook/scripts deployments
![Page 16: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/16.jpg)
Tragedy 2: Run notebook/script as it is using cron
![Page 17: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/17.jpg)
© Daniel Tunkelang - Where should you put your data scientists? - www.slideshare.net/dtunkelang/where-should-you-put-your-data-scientists
Step 1 (management): Integrate data scientists into cross-functional teams
![Page 18: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/18.jpg)
Step 2: Build/Deploy functions, not notebooks
![Page 19: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/19.jpg)
Step 3: Monitor ML in production with other ML
● Data pipeline statistics
● Anomaly detection
● Pattern recognition
● Keep Data Scientist in
the loop
● Treat data errors as
Software bugs
![Page 20: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/20.jpg)
Data Pipeline Functions
![Page 21: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/21.jpg)
Batch Prediction Functions
![Page 22: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/22.jpg)
From Vanilla Spark to serverless training and data processing
./bin/spark-submit
- Spark Sessions Pool
- Functions Registry
- Multi-tenancy
- REST API Framework
- Data API Framework
- Infrastructure
Integration (EMR,
Hortonworks, etc)
![Page 23: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/23.jpg)
![Page 24: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/24.jpg)
UX: Deploy Spark functions and trigger it from apps
![Page 25: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/25.jpg)
Mist - Serverless proxy for Spark
DEMO
![Page 26: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/26.jpg)
Machine Learning: training + serving
![Page 27: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/27.jpg)
pipeline
Training (Estimation) pipeline
trainpreprocess preprocess
![Page 28: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/28.jpg)
pipeline
Prediction Pipeline
preprocess preprocess
![Page 29: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/29.jpg)
cluster
datamodel
data scientist
web app
docker
API
libs
model
Local Spark ML Serving Library:https://github.com/Hydrospheredata/spark-ml-serving
![Page 30: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/30.jpg)
Model Artifact
![Page 31: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/31.jpg)
Models - Runtimes - Formats Zoo
![Page 32: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/32.jpg)
API & Logistics
- HTTP/1.1, HTTP/2, gRPC
- Kafka, Flink, Kinesis
- Protobuf, Avro
- Service Discovery
- Pipelining
- Tracing
- Monitoring
- Autoscaling
- Versioning
- A/B, Canary
- Testing
- CPU, GPU
![Page 33: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/33.jpg)
Sidecar Architecture
![Page 34: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/34.jpg)
![Page 35: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/35.jpg)
UX: Train anywhere and deploy as a Function
![Page 36: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/36.jpg)
UX: Models and Applications
Applications provide public endpoints for the models
and compositions of the models.
![Page 37: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/37.jpg)
UX: Streaming Applications + Batching
![Page 38: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/38.jpg)
UX: Pipelines, Assembles and BestSLA Applications
![Page 39: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/39.jpg)
ML Function as a Service Demo!!!
![Page 40: Serverless machine learning operations](https://reader031.fdocuments.us/reader031/viewer/2022022415/5a647b817f8b9a2c568b4b01/html5/thumbnails/40.jpg)
Thank you
Looking for
- Feedback
- Advisors, mentors &
partners
- Pilots and early adopters
Stay in touch
- @hydrospheredata
- https://github.com/Hydrospheredata
- http://hydrosphere.io/