Disrupt the static nature of BI with Predictive Anomaly Detection
-
Upload
anodot -
Category
Technology
-
view
152 -
download
1
Transcript of Disrupt the static nature of BI with Predictive Anomaly Detection
![Page 1: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/1.jpg)
![Page 2: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/2.jpg)
Disrupt the static nature of BI with Predictive Anomaly Detection
by Nir KalishSenior Director, Solution Engineering, Anodot
![Page 3: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/3.jpg)
3
What is the problem?
Delayed business insights cost companies millions of dollars
Real Time Business Incident Detection
![Page 4: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/4.jpg)
4
Business Incidents Use Cases
Drop in transactions across partners, mobile devices, type of loan
Online Payments/Loans Peak in revenues, impressions across publisher, advertiser, campaign
Ad-Tech
Drop in number of rides across city’s mobile devices, drivers
Ride SharingDrop in conversion for hotel reservations across browsers, countries
E-commerce
![Page 5: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/5.jpg)
5
Getting Business Insights using traditional BI tools? Monitoring Systems?
Maintenance, not
automated
False Positive
No Real time
Millions of metrics
%
0 1 0 1 1 0 1 0 1 0 1 0
![Page 6: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/6.jpg)
6
Find the Anomaly… Using traditional BI tools
Dashboards
![Page 7: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/7.jpg)
7
So How do we get Real Time Business Insight?
![Page 8: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/8.jpg)
8
How to Track the Millions and Get the Insights?
Automated Anomaly DetectionDisrupting the static nature of BI
Aggregate, Detect, Group and Alert
![Page 9: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/9.jpg)
9
What is Anomaly Detection?
![Page 10: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/10.jpg)
10
Automatic Anomaly Detection in five Steps
Metrics Collection – Universal, scale to millions
Normal behavior learning
Abnormal behavior learning
Behavioral Topology Learning
Feedback Based
Learning
1 2 3 4 5
![Page 11: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/11.jpg)
11
Automatic Anomaly Detection in five Steps
Metrics Collection – Granular, scale to millions
Normal behavior learning
Abnormal behavior learning
Behavioral Topology Learning
Feedback Based
Learning
1 2 3 4 5
![Page 12: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/12.jpg)
12
Anomaly Detection in every granularity
Number of Purchases
Product Category Geo Device
OSRevenue $ gift card
TV modelPhone model
Gift cards
Cell Phones
Electronics
US
EMEA
APJ
iOS
Windows
Android
![Page 13: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/13.jpg)
Large Scale Anomaly Detection System Architecture
Kafka
Events Queue
AnomalyGrouping
Signals Correlation
Map
Real-TimeRollups Store
Cassandra
Anodotd
RESTWebApp
OnlineBase LineLearning
Aggregator
Elasticsearch
DWH S3
HADOOP/SparkHIVE
Offline Learning
Management &
Portal
Anodot-Web
User MgmtRDBMS
Customer DSAgent
• 5.4 billion daily samples• 120,000,000 metrics• 240,000,000 models
• updated with each sample• 500,000,000 correlation links
• Updated daily• 14,000,000 seasonal models
• Updated daily• 30 types of learning algorithms
• Metric classification, seasonality detection, trend, baseline models, clustering algos, LSH, …
• And counting…
![Page 14: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/14.jpg)
14
Automatic Anomaly Detection in five Steps
Metrics Collection – Universal, scale to millions
Normal behavior learning
Abnormal behavior learning
Behavioral Topology Learning
Feedback based
learning
1 2 3 4 5
![Page 15: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/15.jpg)
15
Static Thresholds versus Anomaly Based Alert
Anomaly Based Alert will find the problems hours before the static based one
![Page 16: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/16.jpg)
16
Normal Behaviour Modelling: Not so Simple…
Signal TypeSeasonal pattern Adapt to changes
![Page 17: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/17.jpg)
17
Seasonality
![Page 18: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/18.jpg)
18
Learning the normal behavior: Not all signals are created equal
Smooth Irregular sampling
Multi Modal Sparse
Discrete “Step”
Step 1 Classify signal to Category
Step 2Match
Category with
Baseline Distribution
and Algorithm
![Page 19: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/19.jpg)
19
Metric types distribution
![Page 20: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/20.jpg)
20
Distribution of metric types per industry
![Page 21: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/21.jpg)
21
Update rate with adaptive online models: Avoiding pitfalls
What should be the learning rate?
Too Slow
Too Fast
![Page 22: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/22.jpg)
22
Update rate with adaptive online models: Avoiding pitfalls
What should be the learning rate?“Al Dente”
Auto tuning required!
![Page 23: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/23.jpg)
23
Automatic Anomaly Detection in five Steps
Metrics Collection – Universal, scale to millions
Normal behavior learning
Abnormal behavior learning
Behavioral Topology Learning
Feedback Based
Learning
1 2 3 4 5
![Page 24: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/24.jpg)
Abnormal behavior modelP(Anomaly Significance | Duration, Deviation)
90
70
5030 2010
Abnormal Behaviour Learning
![Page 25: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/25.jpg)
25
Abnormal Behaviour Learning
Anomaly Score to enable correct prioritization of problems
![Page 26: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/26.jpg)
26
Automatic Anomaly Detection in five Steps
Metrics Collection – Universal, scale to millions
Normal behavior learning
Abnormal behavior learning
Behavioral Topology Learning
Feedback Based
Learning
1 2 3 4 5
![Page 27: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/27.jpg)
27
Behavioral Topology Learning and Correlation
Viewing correlated metrics in context enables correct problem identification
Price glitch – increase in purchases / decrease in revenue
![Page 28: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/28.jpg)
28
Automatic Anomaly Detection in five Steps
Metrics Collection – Universal, scale to millions
Normal behavior learning
Abnormal behavior learning
Behavioral Topology Learning
Feedback Based
Learning
1 2 3 4 5
![Page 29: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/29.jpg)
29
Feedback Based Learning
Enables continuance improvement of the machine learning process
![Page 30: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/30.jpg)
Weekly anomaly stats: The importance of all steps
Based on 120,000,000 metrics
Normal behavior learning
Abnormal behavior learning
Behavioral Topology Learning
![Page 31: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/31.jpg)
31
Anomaly Detection is every where
Social
Fintech
IT Ad-Tech
E-commerce
IOT
Business Incident
Detection
![Page 32: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/32.jpg)
32
Current Anodot Customers – Partial List
- Pedro Silva, Senior product, Credit Karma
It used to take us up to several days to identify an issue on a specific page, offer, or service that was draining our revenues. Anodot identifies when a metric increases or decreases in real time, so we can resolve it quickly, before business suffers or revenue is lost.
![Page 33: Disrupt the static nature of BI with Predictive Anomaly Detection](https://reader035.fdocuments.us/reader035/viewer/2022070511/58a128d41a28abb91b8b6d99/html5/thumbnails/33.jpg)
THANK YOU