Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s...
Transcript of Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s...
![Page 1: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/1.jpg)
Jason Hiltz-Laforge
Make Haste SlowlyBalancing SRE Diligence in Urgency Driven Organizations
![Page 2: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/2.jpg)
WTF?
• Production Engineering Lead @ Shopify in
• I manage both systems and humans
![Page 3: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/3.jpg)
How fast can the humans go without blowing up the systems too often?
![Page 4: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/4.jpg)
Festina Lente (Make Haste Slowly)
![Page 5: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/5.jpg)
- “Festina Lente”, Wikipedia (https://en.wikipedia.org/wiki/Festina_lente)
“… activities should be performed with a proper balance of urgency and diligence. If tasks are overly rushed, mistakes are made and good long-term results are not achieved.”
![Page 6: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/6.jpg)
VS
![Page 7: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/7.jpg)
Disaster Matrix
![Page 8: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/8.jpg)
Disaster(less) Matrix
![Page 9: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/9.jpg)
Disaster(less) Matrix over Time
![Page 10: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/10.jpg)
Availability Matrix
![Page 11: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/11.jpg)
Availability Testing
![Page 12: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/12.jpg)
Availability Testing
![Page 13: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/13.jpg)
Team Speed
Speed of Iteration
Too Slow Too Fast
![Page 14: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/14.jpg)
Not Enough Haste
Speed of Iteration
Too Slow Too Fast
![Page 15: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/15.jpg)
Not Enough Diligence
Speed of Iteration
Too Slow Too Fast
![Page 16: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/16.jpg)
Just Right
Speed of Iteration
Too Slow Too Fast
Festina Lente
![Page 17: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/17.jpg)
Where’s the Dot?
• Disaster Matrix
• Availability Testing
• Error Budgets
• Outage reports
![Page 18: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/18.jpg)
When the systems do blow up, how do you deal with the humans?
![Page 19: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/19.jpg)
Control over Your Environment
![Page 20: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/20.jpg)
Alerts and Pages
![Page 21: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/21.jpg)
Failure is OK
![Page 22: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/22.jpg)
Root Cause Analysis
![Page 23: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/23.jpg)
How fast can the humans go without blowing up the systems too often? Only as fast as their cost of failure lets them
![Page 24: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/24.jpg)
When the systems do blow up, how do you deal with the humans? By giving them control and safety
![Page 25: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/25.jpg)
Shopify TalksWednesday Standing On the Shoulders of Giants: Unleashing the Power of Scriptable Load Balancers 1:40 pm to 2:30 pm - Emil Stolarsky
Bots Are Fast, Humans Are Smarter—Eliminate Unwanted Traffic and Defend Against DDoS 3:40 pm to 4:10 pm - Felix Glaser
Thursday
Resiliency Testing with Toxiproxy 9:50 am to 10:20 am - Jake Pittis Incident Management and Chatops at Shopify 4:30 pm to 5:00 pm - Daniella Niyonkuru Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo
Friday
Building an On-Premise Kubernetes Cluster For a Large Web Application 11:30 am to 12:00 pm - Daniel Turner
![Page 26: Make Haste Slowly - USENIX · Six Ways a Culture of Communication Strengthens Your Team’s Resiliency (Lightning Talk) 5:00 pm to 6:00 pm - Jaime Woo Friday Building an On-Premise](https://reader033.fdocuments.us/reader033/viewer/2022050600/5fa75edead13bc5c6e1b3473/html5/thumbnails/26.jpg)
https://engineering.shopify.com/ @shopifyeng