The Postmortem – It doesn’t have to be messy
6
© 2013 Blackrock 3 Partners LLC 1 A Good Post Mortem . . . . Includes a Pulse Check! 2 2013 Events…so far 2 You are part of Critical Infrastructure! 3 Postmortem Typical Worst Case (the Usual Case) Yelling & screaming Fighting & biting Firing Everyone hides Blame finding Many resumes updated Stock option date reviewed Nothing is fixed Best Case (Utopia) Process oriented root cause(s) analysis Team effort Focus on positive Focus on prevention Mistakes kept in perspective Completed by non- involved personnel
-
Upload
chef-software-inc -
Category
Technology
-
view
461 -
download
0
description
Your company has had a really, really bad day. Bad Days happen. It is when, not if. In a typical scenario, the top executives come in yelling and screaming and fire whoever they think may have had some part in the Bad Day’s events. After most of the yelling has stopped, then the executives want to know what happened. There is no doubt that autopsies can be messy and emit unpleasant odors, but they are a necessary evil and effective learning tool. But there is a right way and a wrong way to conduct a postmortem analysis of your incident. The obvious objective is to learn what happened so that it doesn’t happen again. The focus should be on the root cause and conducting an efficient analysis of the problem, but there is so much more that can be learned. Using the Incident Command System (ICS), this session provides an overview of the same postmortem process used by Fire Departments across the country.
Transcript of The Postmortem – It doesn’t have to be messy
© 2013 Blackrock 3 Partners LLC 1
A Good Post Mortem . . . .
Includes a Pulse Check!
2
2013 Events…so far
2
You are part of Critical Infrastructure!
3
Postmortem Typical
Worst Case (the Usual Case) Yelling & screaming Fighting & biting Firing Everyone hides Blame finding Many resumes
updated Stock option date
reviewed Nothing is fixed
Best Case (Utopia)
Process oriented root cause(s) analysis
Team effort Focus on positive Focus on prevention Mistakes kept in
perspective Completed by non-
involved personnel
© 2013 Blackrock 3 Partners LLC 2
4
EVENT
A Typical Post Mortem Goes Like This
Failure Operations Process Software Hardware Response Responsibilities
5
Why the Pulse Check?
Cause: Factors that produce an effect Reason: An explanation or justification
Y
6
Looking for “Some Guy”
Failure Operations Process Software Hardware Response Responsibilities
© 2013 Blackrock 3 Partners LLC 3
7
AAR
Marketing/Sales want products faster than Engineering/Operations can design/build/implement them
Investors want faster growth than management can deliver
Early technology architecture/equipment decisions never get replaced
R&D teams love the newest technology Operations teams hate the newest
technology
8
It’s a Matter of Perspective
A wake-up call A near miss for a larger, more
devastating incident An opportunity to stop and look at the
organization – because you never have time to stop and look because the business is running so fast
You don't know what you don't know Don’t let a good crisis go to waste!
9
When done right
The process allows you to look at risk differently
Positive Failure!
© 2013 Blackrock 3 Partners LLC 4
10
TECH Possible vs. Likely
Operations
Copyright 2013 BlackRock 3 Partners LLC
Operations
Normal
Technological
Environmental
Human
Not Normal
Criminal
11
TALENT
Training Accountability Leadership Empowerment Notification Trust/Temptation
Experts or Saviors
12
IMS
Incident Commander
Unified Command
Applications
App 1 App 2
Database
DBA - 1 DBA - 2
Continued Operations
Customer Service
Executive Liaison
Plans
Disaster Recovery
© 2013 Blackrock 3 Partners LLC 5
13
Change – Failure – AAR
Maybe we shouldn’t talk about failure and death together!
Failure
AAR
Change
After Action Review (AAR) Post Incident Evaluation (PIE) Post Incident Analysis (PIA) Incident Review (IR) Reason- Cause – Analysis (RCA)
14
EVENT
Operations Process
Software Hardware
Response
Responsibilities Sales
Marketing
Growth
Technology
Environment
Communication
Human
After Action Review
15
AAR Tradeoff between different groups: Marketing Sales Engineering Operations Finance Management Team Board Investors
© 2013 Blackrock 3 Partners LLC 6
16
Establishing a Culture
Things WILL go wrong There should be a process to address them Incident Management System (IMS)
– Handling of an occurrence – Handling of an issue – Having a really bad day
RCA with a new perspective Retrospective and prospective The key is to understand the reason
17
www.blackrock3.com
Chris Hawley – [email protected]
Rob Schnepp – [email protected]
Ron Vidal – [email protected]
San Francisco & Baltimore
Blackrock 3 Partners LLC
