#USPCA2016: Mushroom Seduction by Bob DiPietro, RI Mushroom Co.
The Mushroom Cloud Effect - What happens when containers fail?
-
Upload
alois-mayr -
Category
Software
-
view
125 -
download
0
Transcript of The Mushroom Cloud Effect - What happens when containers fail?
![Page 1: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/1.jpg)
@mayralois
Mushroom Cloud Effect
Alois MayrTechnology Lead – Dynatrace
![Page 2: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/2.jpg)
@mayralois
Monitoring for Highly DynamicContainer Environments
![Page 3: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/3.jpg)
@mayraloisSource: http://www.schoonoart.de/
…there’s been the mushroom cloud effect
oh yeah, everything screwed up
![Page 4: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/4.jpg)
@mayralois
The Mushroom Cloud Effector
What Happens When Containers Fail?
![Page 5: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/5.jpg)
@mayralois
Biggest LatAm e-commerce Company
• ~ U$ 2.5 billion revenue• 4 sites: Americanas, Shoptime,
Submarino, Soubarato
• ~ 150 hosts across 4 regions• 5k-15k containers• 1k-3k services
![Page 6: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/6.jpg)
@mayralois
TL;DR
![Page 7: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/7.jpg)
@mayralois
About Cloud-Scale Systems
![Page 8: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/8.jpg)
@mayralois
Important Aspects…
• Lots of (micro-)services
• Lots of communication between services
• Service dependencies
• Versioning and API compatibilities
• Zero downtime
![Page 9: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/9.jpg)
@mayralois
Platform-related Aspects
• Most often container-based
• Clustered for scalability
• Ephemeral containers
• Resilient architecture
• Cross AZ fail-overs
• SDN for communication
![Page 10: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/10.jpg)
@mayralois
Deployments are no Longer Static
7:00 a.m.Low load, service runningwith minimum redundancy
12:00 p.m.Scaled up service during peak loadwith failover of problematic node
7:00 p.m.Scaled back down to lower load, move to different geolocation
![Page 11: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/11.jpg)
@mayralois
Anatomy of dynamic environments
https://www.dynatrace.com/en/ruxit/
![Page 12: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/12.jpg)
@mayralois
All About (Service) Dependencies
![Page 13: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/13.jpg)
@mayralois
Failing containers…
…may or may not have an (immediate) impact on service performance
![Page 14: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/14.jpg)
@mayralois
Cascading Failures Lead to aMushroom Cloud Effect
![Page 15: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/15.jpg)
@mayralois
![Page 16: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/16.jpg)
@mayralois
The Hungry Container Breakdown
• Shared /logs partition on host• No log rotation, no archiving for app logs• No proper log management used for Docker environment• Shared /logs partition ran out of space
What was the problem?
![Page 17: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/17.jpg)
@mayralois
The Hungry Container Breakdown
• Container health checks failed• Orchestration killed container and rescheduled new one• Still no free space on /logs• Termination and rescheduling• /var/lib/docker ran out of space• Cluster nodes were no longer able to run any containers
How the problem has evolved over time?
![Page 18: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/18.jpg)
@mayralois
The Hungry Container Breakdown
• Services at the top of the graph• Increased failure rates• Lots of depending Tomcat and DB services affected
How the problem affected services?
![Page 19: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/19.jpg)
@mayralois
![Page 20: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/20.jpg)
@mayralois
The Hungry Container Breakdown
Log management tools for app logs--log-driver=none|syslog
Remove container--rm=true
/var/lib/docker deserves its own partition
How the problem could have been avoided?
![Page 21: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/21.jpg)
@mayralois
The Hungry Container Breakdown
Buggy Containers May Kill Your Nodes
![Page 22: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/22.jpg)
@mayralois
Try to Break Your Clusters Early(And be Prepared for Black Friday)
![Page 23: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/23.jpg)
@mayralois
Break Your Clusters Early
Massive load testing!
Survive three days of pain
Include everythingServices, Containers,
Orchestration, EC2 instances
![Page 24: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/24.jpg)
@mayralois
Testing everything
13.3k containers (+nodes)
3,451 services
![Page 25: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/25.jpg)
@mayralois
![Page 26: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/26.jpg)
@mayralois
Automation Needed to Pinpoint the Root Cause of Cascading Failures!
![Page 27: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/27.jpg)
@mayralois
Want to learn more?
Stop by our booth!G15
![Page 28: The Mushroom Cloud Effect - What happens when containers fail?](https://reader036.fdocuments.us/reader036/viewer/2022081520/58ae28001a28ab557e8b4579/html5/thumbnails/28.jpg)
@mayralois
Thank you!