orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning...
Transcript of orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning...
![Page 1: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/1.jpg)
Demonstration of machine learning function orchestrator (MLFO) via reference implementations
(ITU-ML5G-PS-024)Shagufta Henna, LYIT, 31 July 2020
Register hereJoin us on Slack
OrganizerSponsors
![Page 2: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/2.jpg)
Agenda
• Background
• Functionalities of MLFO
• MLFO Architecture
• Specific concepts: Sequence Diagrams
• Reference Implementation challenge
• Evaluation Criteria
• Participation & Submission Guidelines
• Timeline
2
![Page 3: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/3.jpg)
Background
• Most network operators rely on data scientists to create a ML pipeline
• This ML pipeline if not managed and orchestrated appropriately is subject to bottleneck
3
![Page 4: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/4.jpg)
Background (Cont..)
Specifically, MLFO addresses the following challenges:
• [b-ITU-T Y.ML-IMT2020-Use-Cases] envisages different ML frameworks/ libraries
Example: training of deep learning, tree-based, and linear models require three different ML frameworks
Example: different big data handling tools and libraries, e.g., messaging brokers, data processing engines, etc.
• ML pipeline using these tools/libraries is time-consuming and requires a set of specialized skills
4
![Page 5: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/5.jpg)
Background (Cont..)
• Compatibility issues for integration of [b-ITU T FG-ML5G-I-151] [ML5G-I-203], and [ITU-T Y.3174] need repetitive/complex code
• Lack of standardized ML orchestration mechanism:
• complex/expensive ML pipelines
• handover issues, e.g., bottleneck
• production issues, e.g., glue code, hidden-dependencies, feedback loops, and pipeline nets
• Other challenges for ML pipeline include:
• ML model update, chaining, monitoring, evaluation, pipeline splitting, model deployment, and management &coordination of multiple ML pipeline instances across the network
5
![Page 6: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/6.jpg)
Background (Cont..)
• Uber, Netflix, Google, Facebook, and Airbnb: in-house ML orchestration platforms
• Cloud service providers, e.g., Amazon and Google: ML pipeline orchestration services
• Do not address requirements of diverse use cases [b-ITU-T Y.ML-IMT2020-Use-Cases]
• Do not offer integration with [ML5G-I-203], [b-ITU T FG-ML5G-I-151], and [ITU-T Y.3174] frameworks
• In-house ML orchestration: significant investments & no actual benefits within an industry setting.
6
![Page 7: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/7.jpg)
MLFO Functionalities
Functionalities of the MLFO:
• MLFO can monitor & manage ML pipeline
• Policy-based ML pipeline deployment
• Optimal placement of ML pipeline nodes in the network
• Intent-based specification
• Standard representation & interoperable integration of data handling [ITU-TY.3174], [b-ML5G-I-148], and ML marketplaces [ITU-T Y.ML-IMT2020-MP]
• Chaining/split of ML pipeline nodes, selection of ML models, monitoring modelperformance, reselection and update
• Like NFVO, MLFO decouples ML functions from the underlying network
NOTE- This reduces ML pipeline operational costs with accelerated new offerings
7
![Page 8: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/8.jpg)
Requirements for MLFO
High-level requirements of MLFO:
1. Model Management
2. Data Management
3. ML Pipeline Management
4. Closed Loop System
5. Intent and Policy Management
6. Communication Management
8
![Page 9: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/9.jpg)
1. Model Management
9
ML learning
mechanisms
Training
mechanismsML pipeline
deployment
Automated
testing
ML pipeline
chaining
Model
evaluation
Publish ML
profile
Operations on
ML functions
![Page 10: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/10.jpg)
2. Data Management
10
Data collection
Data
preprocessing Configuration
Associate data
models Synchronization
Data model
mapping
Manage
metadata
Manage data
storage
![Page 11: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/11.jpg)
3. ML Pipeline Management
11
Create
ML pipeline
Delete
ML pipelineModify
ML pipeline
Orchestrate
ML pipeline
nodes
![Page 12: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/12.jpg)
4. Closed Loop System
12
Monitor/OptimizeEnergy efficient
operationResilient
operation
Security &
privacy
Manage &
orchestrate
resources
Model
evaluation in
sandbox
Handle
metering events
ML pipeline
overhead
![Page 13: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/13.jpg)
5. Intent and Policy Management
13
Intent
specification
Operator
specified policies
consultation
Data
retention/deletion
policies
Provision of ML
profile
![Page 14: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/14.jpg)
6. Communication Management
14
Interoperable
knowledge
transfer
Specialized data
formats &
protocols
Subscription of
trained models
Technology
specific
interfaces
![Page 15: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/15.jpg)
15
NF-s : Simulated Network Functions
DM : Data Management Module
MM : Model Management Module
![Page 16: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/16.jpg)
Specific concepts: Data Collection
16
![Page 17: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/17.jpg)
Specific concepts: Data Pre-processing
17
![Page 18: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/18.jpg)
Specific concepts: Model Training
18
![Page 19: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/19.jpg)
Specific concepts: Model Selection
19
![Page 20: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/20.jpg)
Specific concepts: Optimization Flow
20
![Page 21: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/21.jpg)
Specific concepts: Model Deployment
21
![Page 22: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/22.jpg)
Specific concepts: ML pipeline creation
22
![Page 23: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/23.jpg)
Specific concepts: ML Pipeline Testing
23
![Page 24: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/24.jpg)
Specific concepts: ML sandbox-assisted model training/retraining
24
![Page 25: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/25.jpg)
Specific concepts: Model update in marketplace
25
![Page 26: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/26.jpg)
Specific concepts: Asynchronous operation execution
26
![Page 27: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/27.jpg)
MLFO Reference Implementation Challenge
Implementation of specific concepts including:
• Handling ML Intent from operator: a mechanism for operator specify ML use cases via the ML Intent as specified in [ITU-T Y.3172]
• Control of model management, e.g., selection, training and deployment using MLFO
NOTE- No dataset is required for the model management implementation, only meta-data should suffice
• Interaction with ML Marketplace
• Handling of asynchronous operations
• Any other concepts as discussed earlier
27
![Page 28: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/28.jpg)
Our competition schedule is divided into two stages: Phase I and Phase II. These two stages need to submit different competition works.
28
Project ( full marks: 40) Evaluation Standard
Selection of concept demo
(10 marks)
1. Clarity of demo statement
Traceability to ITU-T specifications.
Proof of concept demo plan
Design methodology
(15 marks)
1. Clarity in demo goals
Use case diagram/flow chart
Architecture diagram
Opensource used
Test Setup & Timeline
(15 marks)
1. Details of the test setup
Tracing to requirements and design.
Total 40 marks
Project ( full mark: 60)
Evaluation Standard
Report and PPT
(20 marks)
Detailed report including: i) Demo problem statement, ii) Motivation, iii) Challenges, iv) Milestones achieved, v) Methodology: system design, flow chart, vi) Results and discussion vii) Conclusion
DEMO completion (40 marks)
Demonstratable solution: PoC which maps to the MLFO specification is a must.
Points to take care: Flexibility in possible extensions, potential adaptations and integrations, complete scenario.
Total 60 marks
Evaluation Criteria
Phase IPhase II
![Page 29: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/29.jpg)
Participation & Submission Guidelines
1. Create an ITU account for challenge registration
2. Register for ITU AI/ML in 5G
3. Complete the ITU AI/ML in 5G Challenge Participants Survey with ITU-ML5G-PS-024
4. You can work as an individual or a team of maximum 4 members
5. A GitHub repository should be available shortly to host the code from contestants
All the information here: https://www.lyit.ie/LYIT-ITU-T-AI-Challenge
29
![Page 30: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/30.jpg)
Timeline
Registration Deadline: 21st August 2020
Global Round duration: August -November 2020
Phase I submission: 20th September 2020
Phase II submission: 20th October 2020
Evaluation: October 30th- November 15th
Winners (top 3) official announcement: November 30th
Awards and presentation: December
30
![Page 31: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/31.jpg)
Thank You
Q&A
![Page 32: orchestrator (MLFO) via reference implementations (ITU ......Demonstration of machine learning function orchestrator (MLFO) via reference implementations (ITU-ML5G-PS-024) Shagufta](https://reader034.fdocuments.us/reader034/viewer/2022051915/6006728b216e4410e9795ccc/html5/thumbnails/32.jpg)
AI/ML in 5G Challenge: “Round table + Open house” 07 August 2020
Register hereJoin us on Slack
OrganizerSponsors