Data Center Restoration Incident Action Plan (IAP) Jim... · 2019-11-26 · Data Center Restoration...

Data Center Restoration Incident Action Plan (IAP)Action Plan (IAP)

Powerpoint Templates

Powerpoint Templates

Objectives• Understand the roles and responsibilities for the major

players on the Incident Management Team• Understand the components of the Incident Action Plan

(IAP)(IAP)– 202 (Objectives)– 203 (Organization Assignments)– 204 (Assignments)– 205 (Communications Plan)– 206 (Medical Plan)

Additi l D t (WTX 214 U it L )

Powerpoint TemplatesPage 2

– Additional Documents (WTX, 214 Unit Log)• Understand Span of Control

Prerequisite Information

• IS-700 NIMS an Introduction

• IS-100 Introduction to Incident Command

• IS-200 ICS for Single Resources and Initial Action Incidents

• ICS-300 Intermediate ICS for Expanding Incidents (Class room based training)


Incidents (Class room based training)

3 Types of IAP’s


Mental


Verbal


Written


Incident Command System (ICS) and DR

• ICS was developed from the knowledge th t ff ti lthat no one person can effectively manage all of the facets of a complex, rapidly developing situation.

• The IAP ensures that everyone is working toward the same goals set for that


toward the same goals set for that operational period

NotificationsRecognize andIdentify Problem

Reflex Time

Flow from Event Initiation to Recovery Implementation

Trigger

• 2 am July 3

• Command Center goes dark

• Operators look for a light source to move

• What was that?????

• Get to security office

• Security has no answers

• Got to the data center, it is dark and

• Staff starts calling appropriate persons

• Some are reachable others are


source to move around

dark and quite

others are not

Split Recovery Processes

DeclarationDamage Assessment Complete

Triage

Flow from Event Initiation to Recovery Implementation

Response

p

• Holiday weekend, key persons away

• Travel time to Data

• Appropriate persons start investigating

• We have a BIG problem – IDATF called in

• IDATF starts assessment

• 2 Hours after arrival recommendation is made to IC

• IC reviews with appropriate persons

• DR plan is executed

• Recovery teams go to DR Site

• Data Center Recovery Teams starts


Data Center

starts Recovery Incident Action Plan (IAP)

It’s about the TEAM!

• Characteristics– Work well with others under extreme

circumstances

– Capable of working repeated 12 hours shifts

– Persistent goal driven personalities

– Focus on team goals, not personal gain.


g , p g

– Have a sense of humor

Incident Action Plan• Purpose of the IAP

– Safety of personnel

– Effectiveness of response personnel (Who- What-Effectiveness of response personnel (Who WhatWhen-Where)

• IAP must be accurate enough and complete enough to be a guide for operations

• IAP must constantly be updated, Master during the briefing, updated during the operations


g, p g pperiod

4 Main Elements of the IAP• What do we want to do? (Objectives)

• Who is responsible for doing it? p g(Organization and Assignments)

• How do we communicate with each other? (Communications Plan)

• What do we do if someone is injured? (M di l Pl )


(Medical Plan)

NIMS Forms for the Written IAP• NIMS Forms

– 202 (Objectives)203 (Organization Assignments)– 203 (Organization Assignments)

– 204 (Work Assignments)– 205 (Communication Plan)– 206 (Medical Plan)– Additional Documents (WTX, Maps, 214


Additional Documents (WTX, Maps, 214 Unit Log)

Incident Commander• Responsible to develop SMART objectives (202)

– Specific -Measurable

– Action Oriented -Realistic

Ti S iti– Time Sensitive

• Responsible for choosing team members (203)– Operations Section Chief

– Planning Section Chief

– Logistics Section Chief

Fi Ad i S ti Chi f


– Finance Admin Section Chief

Operations Section Chief• Develops the strategy to meet the

objectives– Strategy is the bridge between objectives andStrategy is the bridge between objectives and

tactics

• Establishes the tactical control points– How will I break this elephant of a problem

down?– Who will I have manage each control point?


g p– This breakdown becomes your 204’s

What the 204 accomplishes• Each Group supervisor knows who works for him/her

and who they report to.• Everyone knows what the operational period is.• Everyone in that group knows how they will• Everyone in that group knows how they will

communicate among themselves and how to call for medical help if needed

• Everyone knows where and when they are to report to• Everyone knows what if any special tools they need to

bring with them• Everyone knows what their works assignments for this


Everyone knows what their works assignments for this operational period are.

Operations Elephant


Breaking the Elephant

• Created three Branch Directors– Infrastructure Branch Director

• With 2 Groups

– Network & SAN Branch Director• With 4 Groups

– Servers Branch Director


• With 5 Groups

Breaking the Elephant

• Infrastructure Branch Director Groups– Electrical Group, responsible for all utility and

emergency power systems including ATS, UPS and PDU’s

– Mechanical Group, responsible for all CRAC’s, support area AHU’s, structural i i l di i d fl


issues including raised floor

Breaking the Elephant• Network and SAN Branch Director Groups

– WAN Group, responsible for all Telco services including AT&T, VzB, and dark fiber, and all internet facing devices

– LAN Group, responsible for all data center routers, switches, physical cabling, rack and stack

– SAN/NAS, responsible for all tiered storage sub-systems

– Voice responsible for all PBX and IVR services


Voice, responsible for all PBX and IVR services

Breaking the Elephant• Servers Branch Director Groups

– Linux Group– VMS GroupVMS Group– Unix Group– Unix Group– Windows Group– AIX/MPE Group

• Each group responsible for all servers


• Each group responsible for all servers within their O/S assignment

Communications Plan 205• Built assuming limited cell service

• Needed to be reliable

• Needed to be simple and license free• Needed to be simple and license free

• FRS –Family Radio Service (UHF Band)– Tested communications from core to property lines

with no adverse issues.

• Command Channel (1-C&G Staff)

• Tactical Channels (3 Branches and Groups)


• Tactical Channels (3 Branches and Groups)

• Medical Channel (Medical Unit Leader or LSC)

Medical Plan 206• We are not working under normal circumstances or

conditions. Items we take for granted need to be documented.

• Working ongoing shifts 7x24 we may have teams from outside the area they are not familiar with our medicaloutside the area, they are not familiar with our medical services

• Where are the medical aid stations, what channel can they be reached on?

• What are the transportation services in the area, ALS or BLS services

• Where are the nearest hospitals? Do they have Trauma centers burn centers and Helipad’s Phone #’s and


centers, burn centers and Helipad s Phone # s and addresses are provided

Additional Documents• Pre-Marked Satellite Site Map from internet based services.

– Mark sides of the building– Base of operation location– Key Infrastructure components, exterior transformers, switchgear, fire

hydrants, service entrances, etchydrants, service entrances, etc• Weather Report

– No internet service, listen to radio based weather broadcasts– How will this forecast affect the tactics during the operational period?

What is the outlook for the next operational period• 214 Unit Log

– Basic format to keep track of actions, agreements, or arguments during the operational period

– Should be dropped off at the end of each operational period at a pre-designated location and then reviewed for situational awareness


designated location, and then reviewed for situational awareness, trending, forecasting, status that may not have made its way back to the ICP by another means.


SPAN OF CONTROL

• Limits the responsibilities and resources being managed by any individual

• Span of Control should be between 3 and 7

• Greater than 7 you are being overloaded

• If you have less than 3 they can probably


be absorbed by the superior position

It’s a Process

• Our first IAP is a very high level with many assumptions.

• Using the process, the plan is constantly updated aligning us with our objectives

til i id t


until incident termination

Where to find the formshttp://www.fema.gov/emergency/nims/JobAids.shtm

http://training.fema.gov/EMIWeb/IS/ICSResource/index.htm


Q ti ?Questions?


Data Center Restoration Incident Action Plan (IAP) Jim... · 2019-11-26 · Data Center Restoration...

Documents

Transcript of Data Center Restoration Incident Action Plan (IAP) Jim... · 2019-11-26 · Data Center Restoration...