Learning the Learner, - Berlin Buzzwords · 1 Learning the Learner, Using Machine Learning to Track...
31
1 Learning the Learner, Using Machine Learning to Track the Performance of Machine Learning Algorithms Ira Cohen, Chief Data Scientist 6 th June, 2016
Transcript of Learning the Learner, - Berlin Buzzwords · 1 Learning the Learner, Using Machine Learning to Track...
1
Learning the Learner,Using Machine Learning to Track the Performance of Machine Learning Algorithms
Ira Cohen, Chief Data Scientist6th June, 2016
2
Machine Learning used to be about publishing papers at NIPS and ICML…
3
The coming of age of Machine Learning…
4
and now it is sexy…
5
The Practical Machine Learning Process
1Define
problem
2Collect and
prepare data
3Train/Test
models
4Deploy to Production
5Track and monitor
6
The Practical Machine Learning Process:The race for automation…
1Define
problem
2Collect and
prepare data
3Train/Test
models
4Deploy to Production
5Track and monitor
7
Lots of tools/platforms/solutions !
8
The Practical Machine Learning Process – The overlooked 5th step…
1Define
problem
2Collect and
prepare data
3Train/Test
models
4Deploy to Production
5Track and monitor
9
Monitoring and Tracking
Step 1 Step 2 Step 3
Define performance metrics and expected
behavior
Collect metrics continuously in production
Track for unexpected changes in metrics
But How to Track and Monitor?Manually is NOT an option…
10
Anomaly Detection
11
Anodot’s Business Incident Detection Platform11
Anodot Platform
Insights
Visualize
Real Time Alerts
Root Cause Investigation
Any Type of Data100% of Data
Data Sources
IOT
Web Business
Telcos
Streaming
Analytics
Machine Learning
Search Engine
Offline Analytics
Banks
12
Detecting Business Incidents: Anomaly Detection
Drop in # of visitors
Decrease in ad conversion on Android Price glitch – increase in purchases / decrease in revenue
13
Anomaly detection: Detecting the Unknowns Saves Time + Money
Industrial IoTIndustrial IoTProactive Maintenance
Detecting issues in factories/machinesProactive Maintenance
Detecting issues in factories/machines
Web ServicesWeb ServicesDetecting business incidents + unknown
business opportunitiesDetecting business incidents + unknown
business opportunities
Machine LearningMachine LearningClosing the “Machine Learning” loop
Tracking and detecting ”unknowns” not modeled during training
Closing the “Machine Learning” loopTracking and detecting ”unknowns” not modeled
during training
SecuritySecurityDetection of unknown breach/attack
patterns Detection of unknown breach/attack
patterns
14
Detecting Unknowns of ML in Production: Anomaly Detection
Classification accuracy over time
15
What is Anomaly Detection?
16
Find the Anomaly
17
Anomaly Detection
17
• Ill posed problem
• What is an anomaly?
18
Anomaly Detection in Time Series Signals
Unexpected change of temporal pattern of one or more time series signals.
19
Anomaly Detection Methods
20
Anomaly Detection: General Scheme
General scheme
Step 1 Step 2 Step 3
Model the normal behavior of the metric(s) using a statistical model
Devise a statistical test to determine if samples are explained by the model.
Apply the test for each sample. Flag as anomaly if it does not pass the test
21
Online Anomaly Detection Algorithms
1
2
3
Initialize model
For each new sample test if anomaly
Update model parameters with each new sample
22
Example Online Models/Algorithms
4
2
1
3
Simple Moving
Average
Double/Triple
exponential (Holt-
Winters)
Kalman Filters +
ARIMA and
variations
Single
exponential
forgetting
23
Batch Anomaly Detection
1 2 3 4
Collect historical samples
Segment samples to similarly behaving segments
Cluster segments according to some similarity measure
Mark as anomalies segments that are in small or no clusters
24
Example Batch Anomaly Detection Methods
Multi-model distributions:• Gaussian models• Generalized
mixture models
One sided SVM
PCA
Clustering methods (K-Means, DBScan, Mean-Shift)MOST COMMON IN USE
Hidden Markov Models
25
Anodot’s Automatic Anomaly Detection
Metrics Collection – Universal, scale to millions
Normal behavior learning
Abnormal behavior learning
Behavioral Topology Learning
Feedback Based
Learning
1 2 3 4 5
26
Large Scale Anomaly Detection System Architecture
Kafka
Events Queue
AnomalyGrouping
Signals Correlation
Map
Real-TimeRollups Store
Cassandra
Anodotd
RESTWebApp
OnlineBase LineLearning
Aggregator
Elasticsearch
DWH S3
HADOOP/SparkHIVE
Offline Learning
Management &
Portal
Anodot-Web
User MgmtRDBMS
Customer DS
Agent
• 3.3 billion daily samples• 75,000,000 metrics• 150,000,000 models
• updated with each sample• 300,000,000 correlation links
• Updated daily• 7,000,000 seasonal models
• Updated daily• 30 types of learning algorithms
• Metric classification, seasonality detection, trend, baseline models, clustering algos, LSH, …
• And counting…
• 3.3 billion daily samples• 75,000,000 metrics• 150,000,000 models
• updated with each sample• 300,000,000 correlation links
• Updated daily• 7,000,000 seasonal models
• Updated daily• 30 types of learning algorithms
• Metric classification, seasonality detection, trend, baseline models, clustering algos, LSH, …
• And counting…
27
Tracking the performance of the algorithms: With Metrics
Anomaly as a result of
deployment
Anomaly quality scoresSeasonality detection distribution
Model switching rate and distribution
28
Alert when results are Abnormal: New type of model required
30
Alert when results are Abnormal:Anomaly in Seasonality Detection Distribution
31
Summary
1Define
problem
2Collect
and prepare
data
3Train/Test models
4Deploy to Productio
n
5Track and
monitor
• Automated anomaly detection• Closing the loop of the
machine learning process• The first step in full automation
of the learning process
