Tribal Learning Analytics R&D Project - SoLAR Storm Presentation
REAL-TIME NETWORK ANALYTICS WITH STORM
description
Transcript of REAL-TIME NETWORK ANALYTICS WITH STORM
![Page 1: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/1.jpg)
REAL-TIME NETWORK ANALYTICS WITH STORM
Mauricio VacasFausto Inestroza
Sonali Parthasarathy
![Page 2: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/2.jpg)
Mauricio VacasBig Data Architect
Sonali ParthasarathyReal-Time Processing
Fausto InestrozaBig Data Architect
Anita MehrotraData Scientist
Susie LuVisualization
Krista SchnellVisualization
Rick DrushalEngineering Lead
John AkredProduct Lead
The Team
![Page 3: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/3.jpg)
WHY REAL-TIME?
![Page 4: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/4.jpg)
Distributed Analytics
Real-Time Data Ingestion
Model Prototyping
Exploratory Analytics
Real-Time Rule Execution
PROCESS
UNDERSTAND
REACT
![Page 5: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/5.jpg)
Accenture Cloud Platform
Recommender as a Service
…
Network Analytics Services
Big Data Platform
![Page 6: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/6.jpg)
Drivers
consumer devices
video usage
Issues
Operational Costs
Understanding service quality degradation
Inefficient capacity planning
![Page 7: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/7.jpg)
INGEST PROCESS
VISUALIZE
ANALYZE
STORE
![Page 8: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/8.jpg)
WHY STORM?
![Page 9: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/9.jpg)
Scalability
Reliability
Data types, size, velocity
Mission critical data
Processing, computation, etc.
Time series / pattern analysis
Fault-tolerance
What do we need?
Multiple use cases
![Page 10: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/10.jpg)
How do we get this from Storm?
Processing guarantees
Low-level Primitives
Parallelization
Robust fail-over strategies
Scalability
Reliability
Fault-tolerance
Processing, computation, etc.
![Page 11: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/11.jpg)
PRIMITIVES
![Page 12: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/12.jpg)
Stream
Spout
Bolt
Topology Suboptimal network speed, geospatial analysis
Request info (IP, user-agent, etc)
Pull messages from distributed queue
Sessionization, speed calculation
Tuple Tuple
![Page 13: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/13.jpg)
PARALLELISM
![Page 14: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/14.jpg)
Nimbus Zookeeper
Supervisor
WT T
WT T
Supervisor
WT T
WT T
![Page 15: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/15.jpg)
Topology
Worker Process
Task
Task
Task
Task
Executor Executor
![Page 16: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/16.jpg)
FAULT TOLERANCE
![Page 17: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/17.jpg)
Nimbus
Supervisor
WT T
WT T
Supervisor
WT T
WT T
Supervisor
WT
W
TTT
TT
TT
![Page 18: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/18.jpg)
RELIABILITY
![Page 19: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/19.jpg)
IP2IP2
IP3
IP1
A
![Page 20: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/20.jpg)
IP2IP2
IP3
IP1
A
![Page 21: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/21.jpg)
SUBOPTIMAL NETWORK SPEED TOPOLOGY AN EXAMPLE
![Page 22: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/22.jpg)
KafkaSpout Pre-process Sessionize
Calculate N/W Speed per Session
Update Speed per
IP
Identify Suboptimal
Speed
Store in Cassandra
Cassandra
Tuple (ip 1) Tuple (ip 1) Tuple (ip 1) Tuple (ip 1) Tuple (ip 1) Tuple (ip 1)Tuple (ip 1)
![Page 23: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/23.jpg)
Cassandra
KafkaSpout Pre-process Sessionize
Calculate N/W Speed per Session
Update Speed per
IP
Identify Suboptimal
Speed
Store in Cassandra
Tuple (ip 2)Tuple (ip 2)Tuple (ip 2)
Tuple (ip 1)Tuple (ip 1)Tuple (ip 1)
Tuple (ip 1)
Parallelism
Tuple (ip 1) Tuple (ip 1) Tuple (ip 1) Tuple (ip 1) Tuple (ip 1)
Tuple (ip 2) Tuple (ip 2) Tuple (ip 2) Tuple (ip 2) Tuple (ip 2) Tuple (ip 2)
![Page 24: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/24.jpg)
Cassandra
KafkaSpout Pre-process Sessionize
Calculate N/W Speed per Session
Update Speed per
IPJoin Compare
SpeedStore in
Cassandra
Speed by Location
Stream 1
Stream 2
KafkaSpout
Tuple (ip 1)
Branching and Joins
Tuple (ip 1/NY) Tuple (ip 1/NY)
Tuple (NY)
![Page 25: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/25.jpg)
RULE EXECUTION
![Page 26: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/26.jpg)
Drools
METHOD 1Storm
METHOD 2Storm + Drools
![Page 27: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/27.jpg)
KafkaSpout Pre-process Sessionize
Calculate N/W Speed per Session
Update Speed per
IP
Identify Suboptimal
Speed
Store in Cassandra
Cassandra
Drools
Storm + Drools
![Page 28: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/28.jpg)
Copyright © 2012 Accenture All rights reserved. 28
Integration with Cassandra
Cassandra Optimal for time series dataNear-linear scalableLow read/write latency
Custom BoltUses Hector API to access CassandraCreates dynamic columns per request Stores relevant network data
![Page 29: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/29.jpg)
Copyright © 2012 Accenture All rights reserved. 29
Lessons Learned
• Rebalance Topology• Tweak Parallelism in bolt• Isolation of Topologies• Use TimeUUIDUtils• Log4j level set to INFO by default
![Page 30: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/30.jpg)
Copyright © 2012 Accenture All rights reserved. 30
DEMO
![Page 31: REAL-TIME NETWORK ANALYTICS WITH STORM](https://reader035.fdocuments.us/reader035/viewer/2022062501/56816711550346895ddb7b32/html5/thumbnails/31.jpg)
Copyright © 2012 Accenture All rights reserved. 31
Next Steps
• Trident• Externalizing Rules • Predictive Models• Real-Time Notifications