Honest Performance Testing with "NDBench" (Vinay Chella, Netflix) | Cassandra Summit 2016
Netflix: A State of Xen - Chaos Monkey & Cassandra
-
Upload
datastax-academy -
Category
Technology
-
view
2.639 -
download
3
Transcript of Netflix: A State of Xen - Chaos Monkey & Cassandra
![Page 1: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/1.jpg)
A State of Xen Chaos Monkey & Cassandra
![Page 2: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/2.jpg)
Who we are
Jean-Sebastien Jeannotte – JS
Senior Software Engineer Platform Automation Engineering
@jsjeannotte
http://www.linkedin.com/in/jsjeannotte
Nir Alfasi
Senior Software Engineer Platform Automation Engineering
@niralfasi
http://www.linkedin.com/in/alfasin
Christos Kalantzis
Director of Engineering Cloud Database Engineering
Cassandra MVP
@chriskalan
http://www.linkedin.com/in/christoskalantzis
![Page 3: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/3.jpg)
AWS Boot re: September 2014, Every AZ
![Page 4: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/4.jpg)
![Page 5: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/5.jpg)
![Page 6: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/6.jpg)
![Page 7: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/7.jpg)
![Page 8: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/8.jpg)
Our stack during Re:boot 2014
C* Priam
C* Priam
C* Priam
REST + SSH
![Page 9: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/9.jpg)
Our stack during Re:boot 2014
![Page 10: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/10.jpg)
Our stack during Re:boot 2014
![Page 11: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/11.jpg)
Our stack during Re:boot 2014
C* Priam
C* Priam
C* Priam
REST + SSH
Atlas Atlas App 1
App 2
![Page 12: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/12.jpg)
Our stack during Re:boot 2014
![Page 13: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/13.jpg)
Our stack during Re:boot 2014
Disappearing instance?
Launch new instance
All good
Is the C* ring healthy?
Yes
Are all instances healthy?
Yes
All good
Can we fix automa>cally?
Replace bad instance
All good
Is there an offline maintenance?
First failure? Sleep for X minutes and
retry
PagerDuty No Is there an offline maintenance?
First failure?
All good
Every 30 min
![Page 14: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/14.jpg)
Our stack during Re:boot 2014
AWS Boot re: September 2014, Every AZ
![Page 15: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/15.jpg)
Gaps we identified
![Page 16: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/16.jpg)
Gaps we identified
![Page 17: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/17.jpg)
Gaps we identified
![Page 18: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/18.jpg)
Gaps we identified
![Page 19: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/19.jpg)
New direction
![Page 20: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/20.jpg)
New direction – What others are doing
![Page 21: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/21.jpg)
New direction – What we decided to do
![Page 22: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/22.jpg)
New direction – What we decided to do
![Page 23: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/23.jpg)
New direction – What we decided to do
C* Priam
C* Priam
C* Priam
Atlas Atlas App 1
App 2
![Page 24: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/24.jpg)
New direction – What we learned (principles)
![Page 25: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/25.jpg)
New direction – What we learned (principles)
![Page 26: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/26.jpg)
New direction – What we learned (principles)
Synchronous Asynchronous
SSH HTTP / REST
![Page 27: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/27.jpg)
New direction – What we learned (principles)
![Page 28: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/28.jpg)
New direction – What we learned (principles)
![Page 29: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/29.jpg)
What does the future look like?
![Page 30: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/30.jpg)
What does the future look like?
![Page 31: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/31.jpg)
What does the future look like?
![Page 32: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/32.jpg)
Check out our https://jobs.netflix.com page for current openings
![Page 33: Netflix: A State of Xen - Chaos Monkey & Cassandra](https://reader031.fdocuments.us/reader031/viewer/2022021813/5875f16b1a28ab006e8b4f91/html5/thumbnails/33.jpg)
Who we are
Jean-Sebastien Jeannotte – JS
Senior Software Engineer Platform Automation Engineering
@jsjeannotte
http://www.linkedin.com/in/jsjeannotte
Nir Alfasi
Senior Software Engineer Platform Automation Engineering
@niralfasi
http://www.linkedin.com/in/alfasin
Christos Kalantzis
Director of Engineering Cloud Database Engineering
Cassandra MVP
@chriskalan
http://www.linkedin.com/in/christoskalantzis