Infrastructure Technologies for Large- Scale Service...
Transcript of Infrastructure Technologies for Large- Scale Service...
![Page 1: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/1.jpg)
Infrastructure Technologies for Large-Scale Service-Oriented Systems
Kostas Magoutis
http://www.cse.uoi.gr/~magoutis
![Page 2: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/2.jpg)
Samza
• Efficient support for state
• Fast failure recovery and job restart
• Reprocessing and lambda-less architecture
• Scalability
![Page 3: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/3.jpg)
General principles: Stateless operators
Courtesy of Borealis Application Programmer’s Guide, Brown Univ. Computer Science Department
![Page 4: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/4.jpg)
General principles: Aggregate operator
Courtesy of Borealis Application Programmer’s Guide, Brown Univ. Computer Science Department
![Page 5: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/5.jpg)
General principles: Join operator
Courtesy of Borealis Application Programmer’s Guide, Brown Univ. Computer Science Department
![Page 6: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/6.jpg)
Stateful processing
Email digestion system (read-only, read-write state)
![Page 7: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/7.jpg)
Stream processing pipeline at LinkedIn
![Page 8: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/8.jpg)
Samza job to find trending tags
![Page 9: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/9.jpg)
Trending tags job
![Page 10: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/10.jpg)
Internal structure of a job
![Page 11: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/11.jpg)
Samza-based applications at LinkedIn
![Page 12: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/12.jpg)
Layout of local state – fault tolerance
![Page 13: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/13.jpg)
Lambda architecture
Samza is lambda-less:• Unified model: treats batch data as finite data stream• Processing late events: Avoid reprocessing entire stream• Reprocessing: leverage Kafka/Databus replaying capability
• Block real-time computation until reprocessing complete• Reprocess in parallel with real-time processing
Accuracy is first class
Latency is first class
![Page 14: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/14.jpg)
Scalable design
• Scaling resources
– Job split into independent and identical tasks
– Input/state partitioning
– Tasks allocated on containers, can be migrated
• Scaling state
– Leverage independent partitioned local stores
– Recovery in parallel across tasks
• Scaling input sources
– Treat each input stream autonomously from other inputs
– Works with variety of systems
• Scaling number of jobs
– No system-wide master
– Jobs are independent, placed on their own set of containers
![Page 15: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/15.jpg)
Checkpointing vs changelog
![Page 16: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/16.jpg)
Checkpointing vs changelog
10 trillion changes/sec
Realistic estimate for change rate
Uncommon in practice
Changelog worse than checkpointing
![Page 17: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/17.jpg)
Experimental setup
• Small (6-node) and large (500-node) clusters
• ReadOnly job (Data enriching, Exception tracing)
– Join between a database and an input stream
• Extract embedded id from each message
• Read (id, val) from database
• Join val with input message
• Output result as new message
• ReadWrite job (EDS, Call graph, Site speed)
– Map ids to counters:
• Extract embedded id from each message
• Read count for id
• Increment counter
• Write counter back
![Page 18: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/18.jpg)
Experimental setup
• Single input stream
• Infinite tuples (id, padding)
– id is randomly generated in range [1, 10k]
– padding is randomly generated string of size m
• k and m are tuning knobs
– k trades off state size for locality
– m is used to tune CPU/network usage
• Choose m such that the system is under stress
– 100 bytes for ReadWrite, 130 bytes for ReadOnly
![Page 19: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/19.jpg)
In-mem vs local disk vs remote disk
Network b/w divided by message size
![Page 20: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/20.jpg)
Network (inbound), CPU utilization
![Page 21: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/21.jpg)
Latency
![Page 22: Infrastructure Technologies for Large- Scale Service ...magoutis/L05/docs/fall17/slides/20171207-samza.pdfSamza is lambda-less: • Unified model: treats batch data as finite data](https://reader035.fdocuments.us/reader035/viewer/2022070719/5edf631dad6a402d666abc80/html5/thumbnails/22.jpg)
Failure recovery