PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach
-
Upload
puppet -
Category
Technology
-
view
101 -
download
0
Transcript of PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach
![Page 1: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/1.jpg)
The Five Dirty Words of CI
J. Paul Reed Release Engineering Approaches PuppetConf, 2017
![Page 2: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/2.jpg)
“CI”
@jpaulreed #PuppetConf
![Page 3: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/3.jpg)
“CI”
@jpaulreed #PuppetConf
![Page 4: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/4.jpg)
@jpaulreed #PuppetConf
![Page 5: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/5.jpg)
J. Paul Reed@jpaulreed on
Alum of The Ship Show
15+ Years in Build/Release Engineering
Now, a DevOps™ Consultant™
Master of Science candidate in Human Factors & System Safety
@jpaulreed #PuppetConf
![Page 6: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/6.jpg)
Root Cause Analysis
Dirty Word #1
@jpaulreed #PuppetConf
![Page 7: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/7.jpg)
@jpaulreed #PuppetConf
![Page 8: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/8.jpg)
Root Cause AnalysisA method of problem solving used for
identifying the root causes of faults or problems.
A factor is considered a root cause if removal thereof from the problem-fault-sequence prevents the final undesirable
event from recurring.— Wikipedia
@jpaulreed #PuppetConf
![Page 9: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/9.jpg)
Our Perception
@jpaulreed #PuppetConf
![Page 10: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/10.jpg)
Our Reality@jpaulreed #PuppetConf
![Page 11: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/11.jpg)
“We found the Root Cause!”
@jpaulreed #PuppetConf
![Page 12: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/12.jpg)
“Root” “Cause”Cause is something
you construct. What you call “root cause”
is simply the place where you stop looking any further.
— Sidney Dekker@jpaulreed #PuppetConf
![Page 13: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/13.jpg)
A Better Choice: “Root Cause Analysis”
Proximate Cause(s)
But…@jpaulreed #PuppetConf
![Page 14: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/14.jpg)
The “Five Whys”
Dirty Word #2
@jpaulreed #PuppetConf
![Page 15: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/15.jpg)
The “Five Whys”Five Whys is an iterative interrogative
technique used to explore the cause-and-effect relationships
underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or
problem by repeating the question "Why?" — Wikipedia@jpaulreed #PuppetConf
![Page 16: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/16.jpg)
What “Five Whys” Always Feels Like to Me@jpaulreed #PuppetConf
![Page 17: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/17.jpg)
The Perception: Incidents are deterministic, like code: same inputs, same outputs.
Every time.@jpaulreed #PuppetConf
![Page 18: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/18.jpg)
A Crash Where?!
int foo(object& r) { r.Blah(); return 1; }
@jpaulreed #PuppetConf
![Page 19: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/19.jpg)
The Operational Reality@jpaulreed #PuppetConf
![Page 20: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/20.jpg)
A Better Choice: “Five Whys”
Just… no. “Swiss Cheese” model
Systemic model
@jpaulreed #PuppetConf
![Page 21: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/21.jpg)
Human Error
Dirty Word #3
@jpaulreed #PuppetConf
![Page 22: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/22.jpg)
Human ErrorHuman error has been cited as a primary cause or contributing factor in disasters and accidents
in industries as diverse as nuclear power, aviation, space exploration, and medicine.
Prevention of human error is generally seen as a major contributor to reliability and safety of
(complex) systems.— Wikipedia@jpaulreed #PuppetConf
![Page 23: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/23.jpg)
But Really: What Is Human Error?
— James Reason’s conception@jpaulreed #PuppetConf
![Page 24: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/24.jpg)
Who gets to draw “the line?” What incentives/interests do they have in putting that “line” where it is? It ignores other stories or even the possibility of entertaining other explanations…
Isues with “Human” “Error”
@jpaulreed #PuppetConf
![Page 25: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/25.jpg)
“Human” “Error”Human error is not the cause of
failure, but the effect.
So, human error… can never be the conclusion of your investigation.
It is the starting point.— Sidney Dekker@jpaulreed #PuppetConf
![Page 26: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/26.jpg)
Human Error often a prelude to a constraint on learning: “Well, just fire the dumb, bad apples… problem solved!”
@jpaulreed #PuppetConf
![Page 27: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/27.jpg)
Were the World So Simple…
@jpaulreed #PuppetConf
![Page 28: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/28.jpg)
A Tiny Problem in “The Cloud”
@jpaulreed #PuppetConf
![Page 29: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/29.jpg)
A Different Take on Failure
@jpaulreed #PuppetConf
![Page 30: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/30.jpg)
A Huge Opportunity to LearnOther operational tools with no input sanity checks The Service Health Dashboard’s real dependencies Indexing Subsystem’s insufficient partitioning Indexing Subsystem hadn’t been fully restarted for years
Had Amazon “just fired” this engineer, they would have never learned these critical details
about their system or how to operate it@jpaulreed #PuppetConf
![Page 31: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/31.jpg)
A Better Choice: Human Error
Stop Saying It. Then, Keep
Not-Saying It.@jpaulreed #PuppetConf
![Page 32: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/32.jpg)
“Why didn’t you?”/ “You should
have…”
Dirty Word #4
@jpaulreed #PuppetConf
![Page 33: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/33.jpg)
Counterfactuals
Dirty Word #4
@jpaulreed #PuppetConf
![Page 34: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/34.jpg)
CounterfactualsCounterfactual thinking is a concept in psychology that
involves the human tendency to create possible alternatives to life events that have already occurred.…
Counterfactual thinking is, as it states, "counter to the facts." These thoughts consist of the "What if?" and the "If I had only..." that occur when thinking of how
things could have turned out differently.— Wikipedia@jpaulreed #PuppetConf
![Page 35: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/35.jpg)
A Waste of Time
@jpaulreed #PuppetConf
![Page 36: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/36.jpg)
Discussing a reality that does not exist.
@jpaulreed #PuppetConf
![Page 37: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/37.jpg)
A Better Choice: Counterfactuals
Don’t.
@jpaulreed #PuppetConf
![Page 38: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/38.jpg)
Best Practice
Dirty Word #5
@jpaulreed #PuppetConf
![Page 39: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/39.jpg)
@jpaulreed #PuppetConf
![Page 40: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/40.jpg)
Best PracticeA best practice is a method or technique
that has been generally accepted as superior to any alternatives because it produces results that are superior to
those achieved by other means or because it has become a standard way of
doing things.— Wikipedia@jpaulreed #PuppetConf
![Page 41: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/41.jpg)
Only the Best of Practices
@jpaulreed #PuppetConf
![Page 42: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/42.jpg)
Only the Best, Artisanal of Practices“Best” is superlative Best practices in complex systems often ignore context Best practices are often not completely defined (especially in complex systems)
@jpaulreed #PuppetConf
![Page 43: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/43.jpg)
Only the Best, Artisanal of Practices“Best practice,” by definition, leaves no space for innovation / discovery Relatedly (and maybe worst of all) it discourages experimentation
@jpaulreed #PuppetConf
![Page 44: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/44.jpg)
Best Practicin’ Ourselves Outta Business
@jpaulreed #PuppetConf
![Page 45: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/45.jpg)
Only the Best, Artisanal of Practices“Best practice,” by definition, leaves no space for innovation / discovery Relatedly (and maybe worst of all) it discourages experimentation Best practice applies to a domain little of our work exists in
@jpaulreed #PuppetConf
![Page 46: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/46.jpg)
Obvious
ComplicatedComplex
Chaotic
Disorder
@jpaulreed #PuppetConf
![Page 47: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/47.jpg)
Obvious
ComplicatedComplex
Chaotic
Governing Constraints
Good Practice
Rigid Constraints Best Practice
Lack of Constraints Novel Practice
Enabling Constraints Emergent Practice
Disorder
@jpaulreed #PuppetConf
![Page 48: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/48.jpg)
Obvious
ComplicatedComplex
Chaotic
Governing Constraints
Good Practice
Rigid Constraints Best Practice
Lack of Constraints Novel Practice
Enabling Constraints Emergent Practice
Disorder
@jpaulreed #PuppetConf
![Page 49: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/49.jpg)
@jpaulreed #PuppetConf
![Page 50: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/50.jpg)
A Better Choice: Best Practice
“Good Practice” Or ensure you apply
“best practice” in the correct domain
@jpaulreed #PuppetConf
![Page 51: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/51.jpg)
“Yeeahhh... so what kind of… Continuous Improvement...
would you... say ya do here?”@jpaulreed #PuppetConf
![Page 52: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/52.jpg)
The Path of Continuous Improvement
is not “linear” (Nor is it “one-and-done”)
Takeaway I
@jpaulreed #PuppetConf
![Page 53: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/53.jpg)
Respect Reality
Takeaway II
@jpaulreed #PuppetConf
![Page 54: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/54.jpg)
Treat People Like the Professionals
They Are
Takeaway III
@jpaulreed #PuppetConf
![Page 55: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/55.jpg)
Go Forth and Continuously Improve
J. Paul Reed www.jpaulreed.com
@jpaulreed
www.release-approaches.comSimply Ship. Every Time.@jpaulreed #PuppetConf
![Page 56: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/56.jpg)
@jpaulreed #PuppetConf
![Page 57: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/57.jpg)
Broken Build
Dirty Word #1
@jpaulreed #PuppetConf
![Page 58: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/58.jpg)
“Flappers”
Dirty Word #2
@jpaulreed #PuppetConf
![Page 59: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/59.jpg)
bobs-mac-mini. local
Dirty Word #3
@jpaulreed #PuppetConf
![Page 60: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/60.jpg)
Merge Window
Dirty Word #4
@jpaulreed #PuppetConf
![Page 61: PuppetConf 2017: The Five Dirty Words of CI- J. Paul Reed, Release Engineering Approach](https://reader036.fdocuments.us/reader036/viewer/2022081517/5a647c547f8b9a2c568b4c6b/html5/thumbnails/61.jpg)
Jenkins Build Number
Dirty Word #5
@jpaulreed #PuppetConf