Multi runtime serving pipelines for machine learning
-
Upload
stepan-pushkarev -
Category
Software
-
view
101 -
download
0
Transcript of Multi runtime serving pipelines for machine learning
![Page 1: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/1.jpg)
Multi-Runtime Serving Pipelines
Stepan Pushkarev CTO of Hydrosphere.io
![Page 2: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/2.jpg)
Mission: Accelerate Machine Learning to Production
Opensource Products:- Mist: Serverless proxy for Spark- ML Lambda: ML Function as a Service - Sonar: Data and ML Monitoring
Business Model: Subscription services and hands-on consulting
About
![Page 3: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/3.jpg)
Deployment | Serving | Scoring | Inference
@Nvidia https://www.nvidia.com/en-us/deep-learning-ai/solutions/
![Page 4: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/4.jpg)
From Single Model to Meta Pipelines
![Page 5: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/5.jpg)
Item 1 Item 2
Title Authentic HERMES Bijouterie Fantaisie Selle Clip-On Earrings Silvertone #S1742 E
Auth HERMES Earrings Sellier Clip-on Silver Tone Round $0 Ship 25130490900 S06B
Specs Brand: HERMESSize(cm): W1.8 x H1.8 cm(Approx)Color: SilverSize(inch): W0.7 x H0.7" (Approx)Style: EarringsRank: B
Brand: HermesFastening: Clip-OnStyle: Clip onCountry/Region of Manufacture: UnknownMetal: Silver PlatedMain Color:SilverColor: Silver
Description ... ...
Does this pair describe the same thing?
Product Matching
![Page 6: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/6.jpg)
Model Artifact: Ops perspective
![Page 7: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/7.jpg)
- HTTP/1.1, HTTP/2, gRPC
- Kafka, Flink, Kinesis
- Protobuf, Avro
- Service Discovery
- Pipelining
- Tracing
- Monitoring
- Autoscaling
- Versioning
- A/B, Canary
- Testing
- CPU, GPU
API & Logistics
![Page 8: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/8.jpg)
Monitoring
Shifting experimentation to production
![Page 9: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/9.jpg)
Sidecar Architecture
![Page 10: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/10.jpg)
Functions registry responsible for the model life cycle and all the business logic required to configure models for serving
Mesh of serving runtimes is an actual serving cluster
Infrastructure integration: ECS for AWS, Kubernetes for GCE and on premise
![Page 11: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/11.jpg)
UX: Models and Applications
Applications provide public virtual endpoints for the
models and compositions of the models.
![Page 12: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/12.jpg)
Why Not just one Big Neural Network?
● Not always possible
● Stages could be independent
● Ad-hoc rule based models
● Physics models (e.g. LIDAR)
● Big E2E DL Requires Black
Magic skills
![Page 13: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/13.jpg)
Why Not just one Python script?
● Modularity. Stages could be developed by different teams
● Traceability and Monitoring
● Versioning
● Independent deployment, A/B testing and Canary
● Request Shadowing and other cool stuff
● Could require different ML runtimes (TF, Scikit, Spark
ML, etc)
● We need more microservices :)
![Page 14: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/14.jpg)
Why Not just TF Serving? ● Other ML runtimes (DL4J, Scikit,
Spark ML). Servables are overkill.
● Need better versioning and
immutability (Docker per version)
● Don’t want to deal with state
(model loaded, offloaded, etc)
● Want to re-use microservices stack
(tracing, logging, metrics)
● Need better scalability
![Page 15: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/15.jpg)
Demo
![Page 16: Multi runtime serving pipelines for machine learning](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a647b747f8b9a57568b4863/html5/thumbnails/16.jpg)
Thank you
- @hydrospheredata
- https://github.com/Hydrospheredata
- https://hydrosphere.io/