2014-08-27 | The Twitter Stack (Vienna Scala User Group)
-
Upload
dominik-gruber -
Category
Technology
-
view
606 -
download
1
description
Transcript of 2014-08-27 | The Twitter Stack (Vienna Scala User Group)
![Page 1: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/1.jpg)
The Twitter StackFinagle, Ostrich, Zipkin,…
Dominik Gruber, @the_dom Vienna Scala User Group – Aug. 27, 2014
![Page 2: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/2.jpg)
Dominik Gruber • @the_domTwitter Stack
Agenda• Introduction
• Prominent Adopters
• Finagle
• Ostrich
• Zipkin
• Iago
• Finatra
![Page 3: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/3.jpg)
Dominik Gruber • @the_domTwitter Stack
Introduction
• Twitter started to move off Ruby on Rails in 2010
• Currently still on Scala 2.9
![Page 4: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/4.jpg)
Dominik Gruber • @the_domTwitter Stack
Prominent Adopters• Box
• eBay
• Etsy
• Foursquare
• Gigya
• Nest
• Rdio
• SoundCloud
• Tumblr
![Page 5: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/5.jpg)
Dominik Gruber • @the_domTwitter Stack
![Page 6: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/6.jpg)
Dominik Gruber • @the_domTwitter Stack
Finagle“Finagle is an extensible RPC system for the
JVM, used to construct high-concurrency servers. Finagle implements uniform client
and server APIs for several protocols, and is designed for high performance and
concurrency. Most of Finagle’s code is protocol agnostic, simplifying the
implementation of new protocols.”
![Page 7: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/7.jpg)
Dominik Gruber • @the_domTwitter Stack
Finagle provides…• Connection pools
• Failure detection
• Failover strategies
• Load balancers
• Back-pressure techniques
![Page 8: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/8.jpg)
Dominik Gruber • @the_domTwitter Stack
Finagle…
• publishes standard statistics, logs, and exception reports
• supports distributed tracing across protocols
• optionally uses ZooKeeper for cluster management
• supports common sharding strategies
![Page 9: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/9.jpg)
Dominik Gruber • @the_domTwitter Stack
Modules (Examples)• finagle-http
• finagle-memcached
• finagle-mysql
• finagle-protobuf
• finagle-redis
• finagle-thrift
![Page 10: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/10.jpg)
Dominik Gruber • @the_domTwitter Stack
Basic abstractions
• Future
• Service
• Filter
![Page 11: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/11.jpg)
Dominik Gruber • @the_domTwitter Stack
Ostrich
![Page 12: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/12.jpg)
Dominik Gruber • @the_domTwitter Stack
Ostrich• A stats collector & reporter
• Collect runtime statistics (counters, gauges, metrics, and labels)
• Report those statistics through a simple web interface or into log files
• Similar to Metrics https://github.com/dropwizard/metrics
![Page 13: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/13.jpg)
Dominik Gruber • @the_domTwitter Stack
![Page 14: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/14.jpg)
Dominik Gruber • @the_domTwitter Stack
Zipkin
“Zipkin is a distributed tracing system that helps us gather timing data for all the
disparate services at Twitter. It manages both the collection and lookup of this data through
a Collector and a Query service.”
![Page 15: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/15.jpg)
Dominik Gruber • @the_domTwitter Stack
Zipkin
![Page 16: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/16.jpg)
Dominik Gruber • @the_domTwitter Stack
Zipkin’s Architecture
![Page 17: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/17.jpg)
Dominik Gruber • @the_domTwitter Stack
Zipkin
• “Scribe” to transport traces to Zipkin and Hadoop
• Data stored in either Cassandra, Redis, HBase, MySQL, PostgreSQL, SQLite, or H2
![Page 18: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/18.jpg)
Dominik Gruber • @the_domTwitter Stack
Iago
![Page 19: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/19.jpg)
Dominik Gruber • @the_domTwitter Stack
Iago
• A load generator that replays production or synthetic traffic
• Iago supports arbitrarily high rates of traffic via built-in support for creating cluster
![Page 20: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/20.jpg)
Dominik Gruber • @the_domTwitter Stack
Protocols• HTTP
• Thrift
• Memcache
• Kestrel (A queueing system from Twitter)
• UDP
![Page 21: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/21.jpg)
Dominik Gruber • @the_domTwitter Stack
Finatra
![Page 22: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/22.jpg)
Dominik Gruber • @the_domTwitter Stack
![Page 23: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/23.jpg)
Dominik Gruber • @the_domTwitter Stack
Features• Fully Async using Futures
• Mustache Templates
• Detailed Metrics (p99,p999, etc)
• Profile/Debug over HTTP
• SSL/TLS Support
• Deploys with a single jar
![Page 24: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/24.jpg)
Dominik Gruber • @the_domTwitter Stack
There’s more…
•
• Summingbird
• https://github.com/twitter
![Page 25: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/25.jpg)
Dominik Gruber • @the_domTwitter Stack
Q & A
![Page 26: 2014-08-27 | The Twitter Stack (Vienna Scala User Group)](https://reader034.fdocuments.us/reader034/viewer/2022042614/55858acbd8b42ae41d8b51ca/html5/thumbnails/26.jpg)
Dominik Gruber • @the_domTwitter Stack
Source• http://blog.oskarsson.nu/post/40196324612/the-
twitter-stack https://blog.twitter.com/2011/finagle-a-protocol-agnostic-rpc-system
• https://www.youtube.com/watch?v=whuT_-cFb5Ihttps://www.youtube.com/watch?v=QdER-Zl_YCM
• https://github.com/twitter https://github.com/finagle