The Hive Think Tank: "Stream Processing Systems" by Karthik Ramasamy of Twitter
-
Upload
the-hive -
Category
Data & Analytics
-
view
353 -
download
0
Transcript of The Hive Think Tank: "Stream Processing Systems" by Karthik Ramasamy of Twitter
![Page 1: The Hive Think Tank: "Stream Processing Systems" by Karthik Ramasamy of Twitter](https://reader036.fdocuments.us/reader036/viewer/2022070512/589c46e71a28ab227d8b502d/html5/thumbnails/1.jpg)
Stream Processing Systems
Karthik RamasamyTwitter
@karthikz
![Page 2: The Hive Think Tank: "Stream Processing Systems" by Karthik Ramasamy of Twitter](https://reader036.fdocuments.us/reader036/viewer/2022070512/589c46e71a28ab227d8b502d/html5/thumbnails/2.jpg)
2
Value of Real Time DataIt’s contextual
[1] Courtesy Michael Franklin, BIRTE, 2015.
![Page 3: The Hive Think Tank: "Stream Processing Systems" by Karthik Ramasamy of Twitter](https://reader036.fdocuments.us/reader036/viewer/2022070512/589c46e71a28ab227d8b502d/html5/thumbnails/3.jpg)
3
Heron
Batching of tuplesAmortizing the cost of transferring tuples
Task isolation
Ease of
debug-ability/isolation/profiling
Fully API compatible with StormDirected acyclic graph
Topologies, Spouts and Bolts
Support for back pressureTopologies should self adjustingg
Use of main stream languagesC++, Java and Python
EfficiencyReduce resource consumption G
Design: Goals
![Page 4: The Hive Think Tank: "Stream Processing Systems" by Karthik Ramasamy of Twitter](https://reader036.fdocuments.us/reader036/viewer/2022070512/589c46e71a28ab227d8b502d/html5/thumbnails/4.jpg)
4
Better Storm
Twitter Heron
Container Based Architecture\Separate Monitoring and Scheduling-Simplified Execution Model2Much Better Performance
![Page 5: The Hive Think Tank: "Stream Processing Systems" by Karthik Ramasamy of Twitter](https://reader036.fdocuments.us/reader036/viewer/2022070512/589c46e71a28ab227d8b502d/html5/thumbnails/5.jpg)
5
HeronSample Topologies
![Page 6: The Hive Think Tank: "Stream Processing Systems" by Karthik Ramasamy of Twitter](https://reader036.fdocuments.us/reader036/viewer/2022070512/589c46e71a28ab227d8b502d/html5/thumbnails/6.jpg)
6
Heron@TwitterStorm is decommissioned
LARG
EST
CLUS
TER
100’
s of T
OPO
LOGI
ES
BILL
IONS
OF M
ESSA
GES
100’s
OF T
ERAB
YTES
REDU
CED
INCI
DENT
S
GOO
D N
IGHT
SLE
EP
3X reduction in resource usage
![Page 7: The Hive Think Tank: "Stream Processing Systems" by Karthik Ramasamy of Twitter](https://reader036.fdocuments.us/reader036/viewer/2022070512/589c46e71a28ab227d8b502d/html5/thumbnails/7.jpg)
Auto scaling the system in the presence of unpredictability
7
Technology Challenges
The Road Ahead
Auto tuning of real time analytics jobs/queries
Exploiting faster networks for efficiently moving data
ÄÜ
J
![Page 8: The Hive Think Tank: "Stream Processing Systems" by Karthik Ramasamy of Twitter](https://reader036.fdocuments.us/reader036/viewer/2022070512/589c46e71a28ab227d8b502d/html5/thumbnails/8.jpg)
8
@karthikz Get in Touch