1

Disaster Recovery / Business ContinuanceDisaster Recovery / Business ContinuanceSept 11 Lessons LearnedSept 11 Lessons Learned

2

IntroductionIntroduction

Material comes from first hand experiences— 25 EMC Customers Affected on Sept. 11

All references to people and companies have been removed except:— Citigroup, Deutsche Bank, JP Morgan, Commerz Bank

Lessons Are Grouped By:— Business Resumption— Application Recovery— Data Recovery

Conclude with summary of EMC’s emergency response

3

Recent Lessons Learned-SummaryRecent Lessons Learned-Summary

Family First then Business!

Data Mirroring Technology worked!— SRDF failover was 100% successful— BUT…people and processes were not always in place to ensure

application recovery and business resumption

Vendor support personnel and capabilities are extremely important to a successful recovery

— In a crisis vendor support staff are critical— Be sure to include them in building and testing your recovery

plans

4

Summary-cont’dSummary-cont’d

Don’t forget to plan for ALL three major types of threats:

— Physical— Natural disaster— Electronic

Be sure your plans cover all three major aspects:— Data Recovery— Application Recovery— Business Resumption

Never Enough Testing and Documentation

5

Business ResumptionBusiness Resumption

Process for declaring Disaster— Need clear chain of command (plus offsite

alternatives)— Predetermine rules and goals

Press Relations— Assign someone to speak on behalf of the

company. People will need information about their family, friends, jobs and state of the company.

6

Business ResumptionBusiness Resumption

Communications— Bandwidth will become especially taxed. Cell phones

clogged, landlines busy. Don't depend on solid communications during a disaster.

— Two way radio communications - the walkie-talkie cell phones would assist DR operations.

— Satellite/cell phones may support communications outside the effected area

— Backup web site for employee communications

Command Post— The availability, location and access by people to the

operations command post is critical.

7

Business ResumptionBusiness Resumption

Airports— Any part of your DR process that includes access to air

travel will be effected. People and tapes expecting to catch flights bound for DR sites were delayed by days. This will also effect the Supply Chain as well.

Intellectual Capital— Protect your people and information.— Executive management has become acutely aware of their

fiduciary obligations to protect assets on behalf of the shareholders and employees.

— Corporate insurance will be reviewed to include coverage for violent acts and coverage for anyone else on the company premises.

8

Business ResumptionBusiness Resumption Disperse your DR support teams

— 2 companies lost their entire DR staff in the attack and were less able to recover operations elsewhere as a result

Loss of Worksite— People need a place to contribute to the

recovery and they will also need a command post too. Size of the facility is important too.

9

Business ResumptionBusiness Resumption Supply Chain Impact

— Airports and transportation systems will impact suppliers, vendors and delivery

— Supply Chain process must be re-adjusted for something other than JIT delivery.

Payroll— People will need access to their wage to cover very

immediate needs. Food, Shelter, clothing etc.

Automate the business processes— If the people aren't able to operate the business, then the

more automation built into the business infrastructure the more able the company will be able to recover, re-synchronize and continue operations. The intellectual capital to run the business may not be available.

10

Application RecoveryApplication Recovery

Business Continuance and Disaster Recovery (BCP) should be a part of the Applications Development and Deployment process.— If left to the LOB manager or IT staff then the

application may not have been included in the DR plan or possess any continuity attributes. BCP should added to the Project Life Cycle process.

E-mail is critical— With the loss of communications and the

workplace, workers will need to work from home. Without e-mail systems they were idle.

— Important for both internal AND external communications to customers, suppliers and even the press.

11

Data RecoveryData Recovery

Replacement of the Primary Site— Make arrangements for a new recovery or

primary facility. Running unprotected during this time is a big risk factor

Back-up the lap-tops and PCs— Many production jobs are in fact run on PCs (large

spreadsheets) for many financial/actuarial applications.

— These were also needed for the DR procedures to be able to recover.

— And they weren't backed-up

12

Data RecoveryData Recovery

All of the Paper Records have disappeared— Strong visual for me on this one, dust clouds swirling

around tons of paper. In addition to the loss of many electronic forms of data, the paper source/back-up will also be effected.

Mirror more data— The cost to add the "other" applications is in retrospect, minimal

and well worth the investment.— While many companies replicated data in real time and it

was very successful. The impact of relying on tape (other sources too) to restore the "lesser" applications negatively impacted the time to recover.

— "Lesser" Applications that were integrated into larger Enterprise Business Applications are now by default mission critical too.

— Sharing information among applications, platforms and networks is an important consideration for recovery

13

Data RecoveryData Recovery

First Come First Served Access to Vendor Recovery Centers— Lack of access to the favorite local, primary

Recovery Sites will force an unexpected travel event. Any specialized equipment may also need to be re-located to new recovery site, further delaying the recovery

Test Frequently— An excellent infrastructure mismanaged or ill

operated is not much better than a poorly designed process.

— Operational Familiarity is key, the more people trained and dispersed the better the plan

14

Data RecoveryData Recovery

A Change management system integrated into the DR process will ensure that recovery included the last change.

Asset management becomes critical because you'll need to know what you have installed and how it's configured. Changing configurations should be included in change management.

15

EMC Emergency ResponseEMC Emergency Response

Established dedicated crisis support center— 24 X 7 in Hopkinton, affected areas got immediate escalation— Extra people and parts dispatched to NJ

EMC facilities offered for customer recovery

Emergency product shipments— Used customer config DB to configure replacement units— 1st replacement shipped and installed night of Sept 11!— Additional 13 units shipped and delivered on Sept 12

16

Closing Slide