Failure as Success Devops Roadtrip Seattle 2016
-
Upload
victorops -
Category
Technology
-
view
219 -
download
0
Transcript of Failure as Success Devops Roadtrip Seattle 2016
![Page 1: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/1.jpg)
‘Failure’ as ‘Success’: The Mindset, The Methods, and The Land Mines
J. Paul Reed / Release Engineering Approaches
VictorOps DevOps Roadtrip 2016, Seattle
![Page 2: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/2.jpg)
Get a Buddy!
#DevOpsRoadTrip @jpaulreed
![Page 3: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/3.jpg)
Get Some Paper!
#DevOpsRoadTrip @jpaulreed
![Page 4: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/4.jpg)
Thought Experiment: An Incident in the JFK “Datacenter”
Dev OpsFrom: Is Your Team Instrument Rated?
vimeo.com/69081899#DevOpsRoadTrip @jpaulreed
![Page 5: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/5.jpg)
#DevOpsRoadTrip @jpaulreed
![Page 6: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/6.jpg)
A Question
Was this event a Success
or a Failure?
#DevOpsRoadTrip @jpaulreed
![Page 7: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/7.jpg)
J. Paul Reed
✤ @jpaulreed on
✤ Alum of The Ship Show
✤ 15 Years in Build/Release Engineering
✤ Now, a DevOps Consultant™
✤ A Master of Science candidate in Human Factors & Systems Safety
#DevOpsRoadTrip @jpaulreed
![Page 8: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/8.jpg)
“Tell Me About Your Feelings About Failure…”
#DevOpsRoadTrip @jpaulreed
![Page 9: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/9.jpg)
#DevOpsRoadTrip @jpaulreed
![Page 10: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/10.jpg)
#DevOpsRoadTrip @jpaulreed
![Page 11: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/11.jpg)
#DevOpsRoadTrip @jpaulreed
![Page 12: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/12.jpg)
‘Failure’ as ‘Success’: The
Mindset#DevOpsRoadTrip @jpaulreed
![Page 13: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/13.jpg)
Success
Failure
Accident
Human Error
Incident
BlameRetrospective
#DevOpsRoadTrip @jpaulreed
![Page 14: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/14.jpg)
Success
Failure
Accident
Safety
Human Error
Incident
BlameRetrospective
#DevOpsRoadTrip @jpaulreed
![Page 15: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/15.jpg)
Safety as “Energy & Barriers”
✤ “Failure” as a release of energy
✤ “Design” safety in
✤ “Defense in depth”
#DevOpsRoadTrip @jpaulreed
![Page 16: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/16.jpg)
#DevOpsRoadTrip @jpaulreed
![Page 17: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/17.jpg)
Safety Impossible Due to “Normal Accidents”✤ Degree of linearity vs.
complexity
✤ Degree of system “coupling”
✤ Certain systems—complex, tightly coupled ones—will have accidents
✤ This is “normal.”(Also? Unavoidable.)
#DevOpsRoadTrip @jpaulreed
![Page 18: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/18.jpg)
Complexity + Tight Coupling =‘Normal’ Disasters
#DevOpsRoadTrip @jpaulreed
![Page 19: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/19.jpg)
Safety in “High Reliability Orgs”
✤ Constant, active learning
✤ Decentralized, active review
✤ Rank deemphasized (oddly?)
✤ Crew rotation
✤ “Success” may be failure
#DevOpsRoadTrip @jpaulreed
![Page 20: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/20.jpg)
Energy & Barriers
“Normal” Accidents
High Reliability Orgs
#DevOpsRoadTrip @jpaulreed
![Page 21: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/21.jpg)
Energy & Barriers
“Normal” Accidents
High Reliability Orgs
Static Processes & (Repeated) Defenses
Active Defenses & Processes
#DevOpsRoadTrip @jpaulreed
![Page 22: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/22.jpg)
Energy & Barriers
“Normal” Accidents
High Reliability Orgs
Technical / Engineering Solutions
Solutions Inherent to People, the Org, the
Way of Working
#DevOpsRoadTrip @jpaulreed
![Page 23: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/23.jpg)
Energy & Barriers
“Normal” Accidents
High Reliability Orgs
Backups, Redundancy,
DR sites
Drilled Incident Response,
Retrospectives, Red-teaming, Value Streams
“Simplify,” Decompose,
Microservices
#DevOpsRoadTrip @jpaulreed
![Page 24: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/24.jpg)
‘Failure’ as ‘Success’: The
Methods#DevOpsRoadTrip @jpaulreed
![Page 25: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/25.jpg)
Successful Failure: The Methods
✤ Inject Failure
#DevOpsRoadTrip @jpaulreed
![Page 26: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/26.jpg)
#DevOpsRoadTrip @jpaulreed
![Page 27: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/27.jpg)
Successful Failure: The Methods
✤ Inject Failure
✤ Incident Command & Crews
#DevOpsRoadTrip @jpaulreed
![Page 28: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/28.jpg)
Successful Failure: The Methods
✤ Inject Failure
✤ Incident Command& Crews
✤ Postmortems!
#DevOpsRoadTrip @jpaulreed
![Page 29: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/29.jpg)
Successful Failure: The Methods
✤ Inject Failure
✤ Incident Command& Crews
✤ (Blameless?)Postmortems!
#DevOpsRoadTrip @jpaulreed
![Page 30: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/30.jpg)
Blameless?
http://jpaulreed.com/blame-aware-postmortems#DevOpsRoadTrip @jpaulreed
![Page 31: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/31.jpg)
Debrief the Actors
#DevOpsRoadTrip @jpaulreed
![Page 32: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/32.jpg)
Gather the Data
#DevOpsRoadTrip @jpaulreed
![Page 33: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/33.jpg)
Create a Timeline
–The Field Guide to Understanding Human Error#DevOpsRoadTrip @jpaulreed
![Page 34: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/34.jpg)
Let’s Create a Timeline!
Write
It On
A Sticky
#DevOpsRoadTrip @jpaulreed
![Page 35: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/35.jpg)
Successful Failure: The Methods
✤ Inject Failure
✤ Incident Command& Crews
✤ (Blameless?) Postmortems!
✤ “Retrospective- ready” Infrastructure
#DevOpsRoadTrip @jpaulreed
![Page 36: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/36.jpg)
Your “Black Boxes”
✤ Chat Ops
✤ TMate
✤ Incident response / management tools
#DevOpsRoadTrip @jpaulreed
![Page 37: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/37.jpg)
‘Failure’ as ‘Success’: The
Land Mines#DevOpsRoadTrip @jpaulreed
![Page 38: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/38.jpg)
Successful Failure: The Land Mines
✤ OrganizationalIncompatibility
#DevOpsRoadTrip @jpaulreed
![Page 39: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/39.jpg)
The best single question for testing an organization’s character is:
What happens when people make mistakes?
— Robert I. Sutton
#DevOpsRoadTrip @jpaulreed
![Page 40: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/40.jpg)
Successful Failure: The Land Mines
✤ Organizational Incompatibility
✤ Only Certain Groups“Get” to Fail
#DevOpsRoadTrip @jpaulreed
![Page 41: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/41.jpg)
An Antipattern: Stopping the Line is a “Privilege”
#DevOpsRoadTrip @jpaulreed
![Page 42: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/42.jpg)
Successful Failure: The Land Mines
✤ Organizational Incompatibility
✤ Only Certain Groups“Get” to Fail
✤ Forgetting to Dampen Failure (Where Possible)
#DevOpsRoadTrip @jpaulreed
![Page 43: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/43.jpg)
Successful Failure: The Land Mines
✤ Only Reviewing Failure
#DevOpsRoadTrip @jpaulreed
![Page 44: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/44.jpg)
#DevOpsRoadTrip @jpaulreed
![Page 45: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/45.jpg)
Successful Failure: The Land Mines
✤ Only Reviewing Failure
✤ Forgetting About Bias
#DevOpsRoadTrip @jpaulreed
![Page 46: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/46.jpg)
A Bit About Biases
Hindsight Bias#DevOpsRoadTrip @jpaulreed
![Page 47: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/47.jpg)
A Bit About Biases
Correspondence Bias#DevOpsRoadTrip @jpaulreed
![Page 48: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/48.jpg)
A Bit About Biases
Correspondence Bias#DevOpsRoadTrip @jpaulreed
![Page 49: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/49.jpg)
A Sea Of Biases
Attention Bias
Backfire Effect
Belief Bias
Confirmation Bias
Congruence Bias Decoy Effect
Contrast Effect
Expectation Bias
Hindsight Effect
IKEA Effect
Impact Bias
Information Bias
Sunk Cost EffectNegativity Effect
Irrational Escalation Effect
Pro-innovation Bias
Optimism Bias
Omission Bias
Rhyme-as-reason Effect
Social Comparison Bias
Social Desirability Bias
Time-Saving Bias
Survivorship Bias
Zero-Risk Bias
Illusion of Truth Effect
Stereotype Bias
#DevOpsRoadTrip @jpaulreed
![Page 50: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/50.jpg)
A Simple, Surprising Example
Bias is built in to the way our brains function.
#DevOpsRoadTrip @jpaulreed
![Page 51: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/51.jpg)
Successful Failure: The Land Mines
✤ Only Reviewing Failure
✤ Forgetting About Bias
✤ De-prioritizing Retrospectives / Learning Processes
#DevOpsRoadTrip @jpaulreed
![Page 52: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/52.jpg)
#DevOpsRoadTrip @jpaulreed
![Page 53: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/53.jpg)
So What Now?
#DevOpsRoadTrip @jpaulreed
![Page 54: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/54.jpg)
Takeaway # 1
Stop thinking about incidents as “events that went wrong.”
Start thinking about incidents in terms of your team’s response.
The key to reframing “failure”:
#DevOpsRoadTrip @jpaulreed
![Page 55: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/55.jpg)
Takeaway # 1
The key to reframing “failure”:Develop Your
“Incident Immune System” (It Gets Better
the More You Use It!)#DevOpsRoadTrip @jpaulreed
![Page 56: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/56.jpg)
Takeaway # 2
To make this practical, your operations and infrastructure
need to be “retrospective-ready.”
#DevOpsRoadTrip @jpaulreed
![Page 57: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/57.jpg)
J. Paul Reed www.jpaulreed.com
@jpaulreed
www.release-approaches.comSimply Ship. Every Time.
![Page 58: Failure as Success Devops Roadtrip Seattle 2016](https://reader037.fdocuments.us/reader037/viewer/2022110219/587b99431a28ab4e4f8b709f/html5/thumbnails/58.jpg)
Adoption of Agile & DevOps methodologies is a journey, and every person and organization’s journey is going to be different.
-J. Paul Reed
DevOps in Practice
J. Paul Reed
http://jpaulreed.com/devops-in-practice