Top 10 Most Common Mistakes During a Crisis...Top 10 Most Common Mistakes During a Crisis Scott Teel...

| 1 www.agilityrecovery.com

Top 10 Most Common Mistakes During a Crisis

Scott Teel Agility Recovery


AGENDA 1. Reliance on a Single Point of Failure 2. Failure to Properly Prepare Employees 3. Too Much Focus on IT Recovery Only 4. Failure to Properly/Sufficiently Insure 5. Over-Reliance on 3rd Party Information 6. Failure to Anticipate Costs of Recovery 7. Failure to Analyze Supply Chain Disruption 8. Failure to Assess the Impact to Customers 9. Failure to Properly Manage Recovery 10. Failure to Test Your Plan


#1 - Reliance on a Single Point of Failure

A. Communications B. Single Recovery Site C. Key Personnel

D. Single Generator E. Single Communicator


A. Communications 1. Network 2. Mobile Devices

• Single Carrier? • Chargers?

3. Single Means of Communication • One email server • Unfamiliarity with texting • Single phone system, with no backup

or means for simple transfer/forwarding • No land lines in the office or at

leadership teams’ homes

SINGLE POINT OF FAILURE

Presenter

Presentation Notes

SO MANY OF US rely on mobile phones - beyond complex little device is a massive network open to all kinds of threats. Same goes for your internal network - Should that go down, applications, data and email communications Having a reliable backup, and alternative means of access to your network, is imperative. One thing we ALWAYS recommend is to diversify carriers Single physical email server - no backup, like Gmail. (cost-effective, simple) ONLY VOIP phones? No traditional land lines? (backup fax line?) - sad reality is this is becoming nearly impossible due to infrastructure in newer buildings Satellite and “burner phones” Zello App Facebook Messenger


B. Single Recovery Site 1. Hot Site? 2. Shared Sites (Shared Branching) 3. Nearby Alternate Facility


Flood Control Infrastructure Issues after Harvey

Erosion, failed concrete, failed pipe, sinkhole, debris, vehicle in drainage channel, dam failure

Presenter

Presentation Notes

Having a plan for a backup site = GOOD ONLY one site and no plan for alternate = BAD work from home strategy Agreement with customers or partners other options like mobile HARVEY in HOUSTON - 600 million square feet of commercial real estate, or 27% of Houston’s total gross leasable area $55 Billion in property value


C. Key Personnel 1. Are any aspects of your plan reliant on a

single individual? 2. Is key information/contacts/passwords held

by a single individual? 3. Is a single person authorized to spend the

necessary funds at the time of disaster?


Bottom Line: If any one individual were unable to report for duty,

would your strategy suffer as a result?

Presenter

Presentation Notes

PEOPLE are a key asset ensure that if any ONE Person is unable to perform their responsibility…you have someone to fill their shoes. internal or external IT manager only one with the knowledge and access credentials Finally – FORCED to make financial decisions during recovery, have multiple people in different locations authorized Cannot rely on employees to remain available


D. Single Permanent Generator 1. Is it protected from the elements? 2. Is it properly serviced & maintained? 3. Do you have service & fuel vendors

arranged? 4. Is it tested regularly? 5. Can it be relocated in an emergency?


Presenter

Presentation Notes

having permanent generator is wonderful to ensure access to power. BUT: fuel is compromised runs out isn’t properly tested and maintained is damaged by the same event while incredibly useful, if you only have one, and isn’t able to be relocated – NOT GOOD


E. Single Communicator/Spokesperson

1. Often the CEO or President 2. Inhibits:

a. Leadership → employee communications b. Decision-making ability c. Media relations d. Organization to stakeholder

communications


Presenter

Presentation Notes

Last on our list of “single points of failure” - having only one person trained or identified as an official spokesperson. NEED a media person – preferably not CEO or President believe it or not. Have quotes instead HAVE at least 2 people identified and a formal strategy outlined on how and when those persons should be contacted.


#2 - Failure to Properly Inform & Prepare Employees

A. Plan Knowledge B. Work from Home Strategy C. Cross-Training

D. Transportation Issues E. Family Preparedness

Presenter

Presentation Notes

Moving on to our SECOND most critical mistake, that is ensuring your employees know WHAT to do, and ensuring they have the ability to DO it.


A. Plan Knowledge

1. Do they Know the plan?

2. Do they know where to access the plan?

3. Do they know their primary role?

4. Have you shared the plan with new hires?

FAILURE TO PREPARE EMPLOYEES

Presenter

Presentation Notes

Plan is worthless unless you’re consistently training people on the plan Walk through it Adjust as needed Engage with new hires Conduct test exercises with team


B. Work from Home Strategy

1. Productivity suffers 2. Inability to login to networks

• Phone/Internet Outages • Power Outages

3. Unwillingness to report to duty • Evacuation Orders • Family or Property in peril

4. Distractions 5. Child Care Issues


●●●●●●●●●● ●●●●●●●●

Presenter

Presentation Notes

BIG TAKEAWAY: Working from home is often the “first line of defense” BUT –should always be considered ONE of several means of recovery - and a temporary one at that. Precocious 4 year old - FAR less productive Problems decreased productivity distractions LOGISTICS (meetings, collaboration, file sharing, applications, etc.) connectivity issues, bandwidth, accessibility problems, network security issues making a major assumption that your employees will actually HAVE power and connectivity at home


C. Cross-Training

1. Critical Functions must continue 2. Certain areas/departments may experience greater demand 3. Longer/Odd Hours may require additional

staffing 4. Consider temporary staff or recently

retired team members as alternates


Presenter

Presentation Notes

no matter WHAT kind of plans and strategies you put into place - some won't be able to respond following a disaster. As you identifying critical functions - take this time to cross-train. MY Example - Retired staff / longer hours / odd hours


D. Transportation Issues

1. Damaged Infrastructure 2. Mass Public Transportation Shut Down

• Car Pooling • Overnight accommodations nearby

3. Fuel Shortages • Storage of Fuel for Critical vehicles/staff • Fuel vendor for deliveries

4. Restricted Access to non-Residents 5. Damaged/Destroyed Vehicles


Presenter

Presentation Notes

Consider Transportation problems Develop a car-pooling plan identifying local hotels plan ahead for low fuel supplies Encourage critical staff to gas up Over three weeks AFTER Florence made landfall, more than 350 roads were still closed in North Carolina including portions of the major North/South interstate of I-95. Implications to SUPPLY CHAIN (FedEx, UPS, USPS)


E. Family Preparedness

1. Do they have a plan? 2. How can your organization help?

• Workshops • Checklists • Emergency Kits

3. Suggested Sites: • Ready.gov • RedCross.org • Do1Thing.org



#3 - Too Much Focus on IT Recovery Only

A. Limitations of Outsourced IT

B. Physical Recovery Elements

Presenter

Presentation Notes

Darned if you do, darned if you don’t Bet that most of you already in place to protect your data and applications BUT – is that where the majority of your planning and investment ends? If so…that’s problem and a common one at that! BIG BOTTLENECK


A. Limitations of Outsourced IT

1. Staff/Time Limitations 2. Responsiveness During Disaster 3. Communications Gaps

TOO MUCH FOCUS ON I.T.

Presenter

Presentation Notes

A lot spend all your time and money on protecting IT environment - BUT don’t think about the HUMAN element Many businesses outsource IT functions to vendors. <<<Automatic flashing red sign>>> They’re great during blue sky days But they have multiple customers regional event…could be stretched across ALL customers have an SOW or statement of work or MOU If you outsource - Determine how to REACH them (AFTER HOURS/WEEKEND/HOLIDAYS – policy and procedure)


B. Physical Recovery Elements

Data Shouldn’t be the Only Recoverable Asset 1. Office Space (meeting rooms, offices, restrooms, etc.) 2. Work Environment (desks, chairs, etc.) 3. Hardware & Equipment (Servers, desktops, laptops, copy, fax) 4. Power & Fuel (Know demand ahead of time)

TOO MUCH FOCUS ON I.T.


#4 - Failure to Properly and/or Sufficiently Insure

A. Coverage Limits/Exclusions B. Lost Revenue C. Added Expenses

D. Flood Insurance E. Contingent Business Interruption

Presenter

Presentation Notes

May be thinking, I’m OK. I’m insured. Same as what thousands of people thought during Sandy, Harvey and Florence


FAILURE TO PROPERLY INSURE

• Ensure you are insured for all potential risks (Can it flood?)

• Consider business interruption insurance and added expense insurance (Protect revenue, but also costs incurred due to interruption)

• Keep photos of your building, equipment lists and policy information stored in a safe and secure offsite location (Can you navigate the claims process quickly/efficiently?)

• Implement an asset management program (Do you know what you have, where it is and how to replace?)

Presenter

Presentation Notes

Know that Insurance only pays, doesn’t physically help LONG process Must be a covered peril Doesn’t cover all expenses Nothing more important during an actual recovery than business interruption insurance and added expense coverage. Have documentation and proof of all your assets, equipment, etc. And ensure you have your policy and contact information stored safely off-site if you have to refer to it and cannot access your facility.


FAILURE TO PROPERLY INSURE

Check Coverage Limits/Exclusions 1. Power Loss (on or off premise) 2. Type of Interruption 3. Type / Cause of Damage

Establish your Operational Downtime Cost - Do the Exercise, establish a cost estimate

Insure Against Lost Revenue - Do you have enough coverage?

Be sure to cover “Added Expenses” 1. Recovery Costs 2. Temporary accommodations 3. Travel Expenses

Presenter

Presentation Notes

TIPS: Check for Utility Coverage Check for exclusions or limits (inventory, revenue loss, reputational damage, liability) Know your cost of downtime Know what it COULD take to recover


#5 – Over-Reliance on 3rd Party Information

A. Power Restoration & Other Utilities B. Permission to Access Facilities C. Delivery of Critical Supplies & Materials


OVER-RELIANCE ON 3RD PARTY INFO

A. Power Restoration & Other Utilities B. Permission to Access Facilities

• Confusion & Territoriality • Know Your Local Officials (Police, Fire, Emerg. Mgmt)

C. Delivery of Critical Supplies/Materials • 72 Hour Rule • Culture of Preparedness • Supply Chain Interruption • Commodity Shortages

Presenter

Presentation Notes

Poor information or misinformation from Utilities Situations where you are told you MAY or MAY NOT access certain areas or facilities - Las Vegas Shooting, Boston Marathon Bombing, or Mexico Beach after hurricane Michael. Have a relationship with your local officials. When will resources be delivered, services like EMS/Fire/Police/Healthcare When will supply chain be restored


#6 – Failure to Anticipate Costs of Recovery

A. Relocation Costs B. Network Recovery C. Commodity Costs/Availability


FAILURE TO ANTICIPATE COSTS OF RECOVERY

A. Relocation Costs • Travel Costs • Accommodations • Family Issues

B. Network Recovery • Bandwidth Issues • Time for Restoration/Recovery of Data • Critical Staff Needs vs. Full Staff Recovery

• Childcare • Per Diem • Remote Network Access (Satellite Connectivity)

Conduct a Recovery Test and do a FULL RESTORE during the exercise to determine duration and potential cost.

Presenter

Presentation Notes

COST to relocate your people or your organization as a whole: travel expenses multiplied by EVERY SINGLE employee plus equipment, etc. (Families) Consider bandwidth and data usage charges Consider limitations of bandwidth (on a plane) Running web applications? Working with large files?, or even fully RESTORE your data to a local machine over a limited bandwidth connection, you may woefully underestimate the time and costs to do so! In these cases, we recommend you VERY CAREFULLY outline what critical staff must remain operational, as well as what functions must remain operational…and establish a plan for recovering those elements, but maybe TIER or PHASE the recovery of less critical elements over time.


FAILURE TO ANTICIPATE COSTS OF RECOVERY

C. Commodity Costs • Fuel Shortages • Rationing • Water/Food distribution

Presenter

Presentation Notes

Fuel Recovery materials, food and water, etc. PETTY CASH!! (How much though) Last on our list is just something we have run into quite a bit in the past 30 days, and something we also saw in the wake of Sandy…and that is the cost of recovering documents. There are some GREAT companies out there that can help you recover and restore paper documents that are critical…but be careful as this is a very costly endeavor. It is a complex and time-consuming, manual process, so don’t drop off dozens of boxes of document for restoration unless you are VERY SURE you need every single one of those pages restored. Otherwise you could run into a bill that is in the 10’s of thousands of dollars. Instead, you should have a digital document storage policy or plan in place, and that could potentially serve you in multiple ways and save you significant costs over the long term.


#7 - Failure to Properly Analyze Supply Chain Disruption

A. Outsourced Payroll Companies B. 3rd Party IT C. Communications/Wireless Vendors

D. Attorney/CPA Firm E. Delivery/Shipping Partners

Presenter

Presentation Notes

As we begin to round out the top 10, we turn our attention to your partners, your suppliers and vendors. A lot of people assume that a supply chain is only relevant if you’re in the manufacturing business. However, every single business on the line today has a supply chain, regardless of what you do. Supply chain actually refers to any vendor, supplier or partner that you rely on to perform your critical business functions.


SUPPLY CHAIN DISRUPTION

A. Outsourced Payroll Companies • Do you know their Recovery Plan? • What is the process for an interruption during Pay Cycle? • Are they integrated into your Exercise?

B. 3rd Party IT: BOTTLENECK C. Communications/Wireless Vendors: BOTTLENECK D. Attorney/CPA Firm

• Can you reach them following in interruption? • Are they involved in your planning process?

E. Delivery/Shipping Partners • What is the protocol for an interruption? • How will any delay from these partners affect your business?

Presenter

Presentation Notes

This should really hit home. Same goes for 3rd Party IT


#8 - Failure to Properly Assess Impact to Customer Base

A. Unable/Unwilling to remain customers

B. Limited/Restricted Access C. Loss of Interest

Presenter

Presentation Notes

UNIQUE - doesn’t have to do with YOUR organization - INSTEAD - what is happening out there in the world


IMPACT TO CUSTOMER BASE

A. Unable/Unwilling to remain Customers • The Show Must Go On…

• Will they Wait? • Can they Wait? • Are your competitors going to pounce?

B. Limited/Restricted Access • Restricted access to your location(s)? • Curfews? • School closings/business closings?

C. Loss of Interest • Following a disaster, will discretionary spending fall off? • When will it return to “business as usual”?

Presenter

Presentation Notes

If your customers are impacted, say they’re forced to evacuate or relocate permanently. Think about the hundreds of thousands of people who left New Orleans after Katrina, and all of those who chose to never return. If those populations formed a large percentage of your customer base…how would you respond? How would you adapt to the situation? If you are forced to relocate your business to the other side of town, or close one location and redirect customers to an alternate site…will they be WILLING to still do business with you or will your competitors step in to serve them? Also, as I mentioned with situations calling for evacuations…if you rely on regular foot traffic in your business, if the customers are forced to leave…what do you do? Do you follow them and set up temporary locations? Do you shut the doors and hope for the best? Even if you think there’s not much you can do to RECOVER from that scenario…do you at least have a plan to reduce expenses, change your go-to-market approach or some other means of at least SURVIVING the time your customer base has vanished? And then finally loss of interest. Say your area experiences a major disaster. People are going to be consumed by recovering their lives, returning to work, assisting their families, fixing or repairing their homes. If your business relies on discretionary income…you could have a MAJOR decrease in revenue. Are you prepared for that kind of change in business?


#9 - Failure to Properly Manage Restoration of Operations

A. “Crush” of Customers B. Inability to Communicate Status C. Operations Restored:

- BUT Employees Not Reporting

D. Facilities Operational/Undamaged: - BUT Supply Chain Disrupted

Presenter

Presentation Notes

Now when it comes to preparing for disasters, preparing for business interruptions and putting plans and strategies into place to help mitigate risks or overcome the actual interruption, an area I’ve personally seen some entities struggle with is the eventual REOPENING of their business…or the restoration period.


RESTORING OPERATIONS

A. “Crush” of Customers • Once the lights come back on…

• Do you provide a high demand service or product? (banks, insurance, healthcare, government)

• Are your services limited during recovery? • Are you working with limited staff?

B. Inability to Communicate Status

• Is your Website operational? • Do you have access to your site to update

status? • Can you answer the phone? • Can you utilize Social Media?

Presenter

Presentation Notes

Some organizations don’t properly plan for the day their doors re-open, or when the customers finally start returning to the business. Are you prepared for a massive influx of customers all at once? Is your team ready for that? Will you have enough staff on hand? Do you have all the necessary supplies and assets on hand to serve your customers? Have you attempted to communicate with your customer base on any limitations you may experience in your ability to deliver goods and services? Speaking of communications…when the time comes to reopen…are you able to communicate that to a large audience? How about updating your website? Or posting to social media? Are your phone lines operational and do you have the ability to change or quickly update outgoing voicemail greetings?


RESTORING OPERATIONS

C. Operations Restored: But employees can’t or won’t report to work.

• Were they prepared at home? • Are schools closed? • Can employees commute safely? • Are Curfews in place preventing travel? • Are areas restricted to homeowners/residents only?

D. Facilities Operational/Undamaged…

But supply chain still disrupted. • Raw materials shortage or transportation/delivery issues • Utilities interruption lingering

Presenter

Presentation Notes

How about your employees themselves? Say Your building is fine. Power is on, your operations can be restarted…but are your employees still stuck at home because of transportation issues, or are their kids still out of school and daycare’s closed? What about any situations that would prevent them from traveling to work? If you’re ready to open your doors, make sure you have been in daily contact with your employees and you’re able to determine IF they can report to work, and do so safely. Finally, going back to our point in Number 7 – is your supply chain prepared to restore services to your organization? Including utilities!


#10 - Failure to TEST Your Plan

A. Test Data Restoration B. Test Alert Notification C. Test Employees’ Knowledge of Plan

D. Test Vendors’ Resilience E. Know Your Power Needs

Presenter

Presentation Notes

Now let’s wrap things up with something I mentioned on the very first slide of today’s presentation. No matter what plan or strategy you put into place…it isn’t worth the paper you printed it on unless you test it. You HAVE to understand how critical it is that you perform regular, planned exercises in order to understand the reality of the plans you’ve put into place, the time it will take to facilitate said plans, and push your strategy to the limits so you can constantly improve. This is a never-ending, ongoing process.


FAILURE TO TEST YOUR PLAN

A. Test Data Restoration • Can you restore with the information at hand? • How long will it take? • Can you recover to new/different hardware? • Do you have access to the necessary software?

B. Test Alert Notification • Can you activate the system remotely? • Can more than one person access the system? • If you utilize a phone tree, is it updated?

Presenter

Presentation Notes

So a few things you can test today. First – as we have mentioned before, you are probably ALL backing up important files, data, applications, etc. BUT – when is the last time you tested actually RESTORING that data to a new machine or to an alternate location? Do you know what you need in order to make that happen? Passwords, applications, the right internet access, etc. Secondly, testing your alert notification system is an easy way to take action today and improve your plan. Do you have a system? Are you using some other form of emergency comms like a phone tree or texting? Do you have multiple people who can develop and send urgent messages and are approved to do so?


FAILURE TO TEST YOUR PLAN

C. Test Employees’ Knowledge of the Plan • What happens if YOU aren’t there? • If the office burns tonight, what is their first step? • Can they access email/text messages/voicemail remotely?

D. Test Vendors’ Resilience • Involve vendors/partners/suppliers in your exercises • Know their recovery plan and be able to integrate it into your own plan

E. Know Your Power Needs • No other takeaway MORE IMPORTANT from the aftermath of Sandy • Caused multiple, complicated, costly delays • A SIMPLE test would have shown this shortcoming for all entities

Presenter

Presentation Notes

The next two have to do with ensuring those you may rely on during a recovery KNOW what they need to do, HAVE the means to do it, and are ABLE to do so in an acceptable amount of time. And the last thing on our list..and the last element of our presentation today


QUESTIONS?

Scott Teel [email protected]

mailto:[email protected]

Top 10 Most Common Mistakes During a Crisis...Top 10 Most Common Mistakes During a Crisis Scott Teel...

Documents

Transcript of Top 10 Most Common Mistakes During a Crisis...Top 10 Most Common Mistakes During a Crisis Scott Teel...