Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded...
Transcript of Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded...
![Page 1: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/1.jpg)
Earthquakes, Forest Fires and Your Next Production Incident
Alex Hidalgo (@ahidalgosre)Squarespace Site Reliability Engineering
![Page 2: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/2.jpg)
Saturday, September 27, 1970
Mount Laguna, California
![Page 3: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/3.jpg)
THE LAGUNA FIRE
◦ Third largest recorded California fire at the time
◦ Spread over 30 miles in just 24 hours
![Page 4: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/4.jpg)
THE LAGUNA FIRE
◦ Impact:▫ 175,425 acres of woodland▫ 382 residences▫ 16 humans
![Page 5: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/5.jpg)
U.S. Department of Agriculture - Flickr: 20130817-FS-UNK-0004
1970 CALIFORNIA FIRE SEASON:576,508 ACRES722 BUILDINGS
> $1.5B IN DAMAGES
![Page 7: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/7.jpg)
The 1970 California fire season was one of the worst on record.
![Page 8: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/8.jpg)
CHANGES IMPACT EVERYTHING
◦ New land-use legislation◦ The Wilderness Act of 1964◦ A growing population
![Page 9: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/9.jpg)
CHANGES EMIT CHANGES
![Page 10: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/10.jpg)
U.S. Department of Agriculture - Flickr: 20130817-FS-UNK-0004
![Page 11: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/11.jpg)
NO GRAY WOLVES
◦ Elk less likely to move around◦ Ate the same willow plants to the
ground◦ Beavers didn’t have their supplies
![Page 12: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/12.jpg)
Thursday, January 12, 1995
Reintroduction of Gray Wolves
![Page 13: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/13.jpg)
WITH GRAY WOLVES
◦ Elk more likely to move around◦ Willows left more intact◦ Beavers have supplies
![Page 14: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/14.jpg)
MORE WOLVES =MORE BEAVERS =WATERSHED CHANGES
![Page 15: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/15.jpg)
BACK TO FOREST FIRES
![Page 16: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/16.jpg)
CHANGES EMIT CHANGES
◦ New land-use legislation◦ The Wilderness Act of 1964◦ A growing population◦ More responding agencies
![Page 17: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/17.jpg)
Fall of 1970
A meeting of the minds
![Page 18: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/18.jpg)
RESPONDING AGENCIES MEET
Problems:◦ Terminology differences◦ Containment techniques◦ Organizational structures◦ Poor communications
![Page 19: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/19.jpg)
92nd Congress of the US, 1971
Funding approved
![Page 20: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/20.jpg)
FIRESCOPE
FIrefightingREsources ofSouthernCaliforniaOrganized forPotentialEmergencies
![Page 21: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/21.jpg)
SERIOUS RESEARCH COMMENCES
◦ Studies!◦ Research!◦ More studies!◦ Collaboration!◦ Feedback from others!
![Page 22: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/22.jpg)
AFTER LOTS OF RESEARCH
Concluded requirements:◦ Formalized communications◦ Formalized hierarchies◦ Formalized response◦ No more freelancing!!!
![Page 23: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/23.jpg)
Research continued for many, many years...
![Page 24: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/24.jpg)
Fire Season, 1978
The Incident Command System is used
![Page 25: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/25.jpg)
SUCCESS!
As time progressed, ICS was adopted for:◦ Other forest fires◦ HAZMAT situations◦ All natural disasters◦ Urban Search & Rescue
![Page 26: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/26.jpg)
Monday, November 25, 2002
ICS is mandatory
![Page 27: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/27.jpg)
“ “Why do I care about any of this?”
![Page 28: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/28.jpg)
The tech industry is hurtling towards adopting known processes instead of continuing to invent our own.
![Page 29: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/29.jpg)
ALMOST NOTHING IS NEW
◦ Engineers have been focused on reliability for as long as humans have been building stuff
◦ Statisticians have been analyzing data for centuries
◦ The ICS has been around for decades
![Page 30: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/30.jpg)
PROBLEMS THE ICS ADDRESSES(that probably apply to you)
◦ Lack of insight◦ Poor communications◦ No established hierarchy◦ Too much freelancing
![Page 31: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/31.jpg)
It’s gotten complicated...
![Page 32: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/32.jpg)
http://www.training.fema.gov/EMIWeb/IS/ICSResource/assets/reviewMaterials.pdf
![Page 33: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/33.jpg)
“ “How can I use the ICS?”
![Page 34: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/34.jpg)
Just one example of how this might work...
![Page 35: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/35.jpg)
A Easy And Effective ICS(for computer things)
![Page 36: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/36.jpg)
INCIDENT COMMANDER
![Page 37: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/37.jpg)
INCIDENT COMMANDER (IC)
◦ In charge of the incident and holds all high-level state about it
◦ This is the only role that must always exist during any incident response
◦ Is responsible for delegating other roles to other engineers
![Page 38: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/38.jpg)
INCIDENT COMMANDER (IC)
◦ If the other roles have not been delegated, it should be assumed the IC is also fulfilling those roles
◦ The role of IC can and perhaps should be handed off
![Page 39: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/39.jpg)
INCIDENT COMMANDER
OPERATIONS LEAD
![Page 40: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/40.jpg)
OPERATIONS LEAD
◦ In charge of making changes to the system in order to mitigate or resolve the problem
◦ No one else should be touching the production besides the OL
◦ Often is the original responder
![Page 41: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/41.jpg)
OPERATIONS LEAD
◦ Actions taken should be documented in a command post
◦ This role is delegated, not free for anyone to pick up
![Page 42: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/42.jpg)
COMMAND POST
◦ Establish a clearly defined communications channel
◦ New or old is fine◦ Text is preferred over voice◦ IC should feel free to police this
channel as much as needed
![Page 43: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/43.jpg)
INCIDENT COMMANDER
OPERATIONS LEAD
COMMUNICATIONS LEAD
![Page 44: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/44.jpg)
COMMUNICATIONS LEAD
◦ Responsible for all communications, both internally and externally
◦ Should be the only one updating things like your status page
![Page 45: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/45.jpg)
COMMUNICATIONS LEAD
◦ May also be a good option for keeping up an Incident State Documents (ISD)
◦ This role is delegated, not free for anyone to pick up
![Page 46: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/46.jpg)
INCIDENT STATE DOCUMENTS
◦ Used to consolidate the current state of the world
◦ Documents which roles have been defined and who currently has them
◦ Templates are good (tooling, too!)
![Page 47: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/47.jpg)
/copy is a neat hack
![Page 48: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/48.jpg)
INCIDENT COMMANDER
OPERATIONS LEAD
COMMUNICATIONS LEAD
PLANNING LEAD
![Page 49: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/49.jpg)
PLANNING LEAD
◦ In charge of supporting the other leads as needed
◦ Others Leads are focused on an immediate fix, the Planning Lead may be focused on future
◦ Responsible for finding new engineers
![Page 50: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/50.jpg)
PLANNING LEAD
◦ Support could extend as far as ordering dinner or fetching coffee
◦ This role is delegated, not free for anyone to pick up
![Page 51: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/51.jpg)
ICS AND FLEXIBILITY
◦ The ICS has been deliberately designed to be able to expand and contract
◦ Make it work for you, don’t just listen to me
![Page 52: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/52.jpg)
An example of expansion
![Page 53: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/53.jpg)
INCIDENT COMMANDER
OPERATIONS LEAD
COMMUNICATIONS LEAD
PLANNING LEAD
![Page 54: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/54.jpg)
OPERATIONS LEAD
DATABASES OPERATIONS
LEADNETWORKING OPERATIONS
LEAD
SYSTEMS OPERATIONS
LEAD
![Page 55: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/55.jpg)
Handing off is important
![Page 56: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/56.jpg)
INCIDENT COMMANDER 1
INCIDENT COMMANDER 2
![Page 57: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/57.jpg)
INCIDENT COMMANDER 2
INCIDENT COMMANDER 1
![Page 58: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/58.jpg)
INCIDENT COMMANDER 2
INCIDENT COMMANDER 1
![Page 59: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/59.jpg)
INCIDENT COMMANDER 2
![Page 60: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/60.jpg)
“ “Am I having an incident?!?”
![Page 61: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/61.jpg)
IS THIS AN INCIDENT?
◦ If you’re wondering if it is, it probably is◦ It’s easier to declare an incident for
something that turns out to be small than it is to apply the framework to an incident after time has passed
![Page 62: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/62.jpg)
IS THIS AN INCIDENT?
◦ Do not try to hide an incident.◦ Stuff breaks! It’s just how it is!
![Page 63: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/63.jpg)
The ICS works… mostly...
![Page 64: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/64.jpg)
2005: HURRICANE KATRINA80% OF NOLA FLOODED
1826 DEATHS> $125B IN DAMAGES
![Page 65: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/65.jpg)
![Page 66: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/66.jpg)
“
“The federal government, in particular the Federal Emergency
Management Agency (FEMA), received widespread criticism for a
slow and ineffective response to Hurricane Katrina. Much of the
criticism is warranted.”
https://www.oig.dhs.gov/assets/Mgmt/OIG_06-32_Mar06.pdf
![Page 67: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/67.jpg)
LOTS WENT WRONG
Final lessons:◦ Anticipate◦ Train◦ Test
![Page 68: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/68.jpg)
ANTICIPATE
◦ Incidents will occur.◦ Make sure your version of the ICS is
ready and documented
![Page 69: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/69.jpg)
TRAIN
◦ Develop workshops and training sessions
◦ Provide tooling and templates◦ Conduct meaningful incident
retrospectives and share them widely
![Page 70: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/70.jpg)
TEST
◦ Run test scenarios◦ Use chaos engineering◦ Operational underload can be as
dangerous as operational overload
![Page 71: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/71.jpg)
Have an Incident Commander
Delegate all roles and enforce them
Establish a command post
Communicate and document incident state
Expand and contract at will
Hand-off regularly
Test and train your processes and procedures
![Page 72: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/72.jpg)
Use the ICS, because, mostly, the ICS works.
![Page 73: Prod uction Incident and Your Nex t Earthquakes, Forest ... · AF TER LOTS OF RESEARCH Concluded requirement ... As time progressed, ICS w as adopted for: Other forest fires HAZMAT](https://reader035.fdocuments.us/reader035/viewer/2022063015/5fd3860652d6244428389015/html5/thumbnails/73.jpg)
Thank you!
Alex Hidalgo - @ahidalgosrealex-hidalgo.com
Shout Outs:Squarespace EngineeringLFI SlackSlidesgala.com