A Case StudyThe Next Step Business

Continuity Workshop

For major Medical Center

Partial Key Performance Indicators for Medical Center’s IT Business Continuity

Title of KPI: Definition: Measurement: Target:

Backup window Duration daily planned application outage

Total time in minutes appl must be quiesced

Time to recover (i.e. time to switch sites)

Time to switch sites and restore service

Total time in minutes: outage => users online

Testing frequency (preparedness)

Frequency per month of end to end BC test

Average system response time

Application response time at bedside

Average and peak response time in sec

Average problem resolution time

Average time identify, resolve applic. problem

Bandwidth costs Monthly cost of bandwidth to DR site

Dollars/month of expense

How much data is being replicated

Amount of data being replicated by PR,Radio

Percentage growth rate data

Annual growth % data production PR, Radio.

Quarter to quarter data allocation report

Pro

cedu

ral

Fin

anci

alStep 1:

Collect InfoFor Prioritization

Scope - Resource and Business Impact

Component Patient Records Radiology E-mail

Applications PR1 EPIC1, PR1 MSE1, MSE2

Data and data management policies

Backup daily at night by TSM

Backup daily by Veritas Backup daily by MSM

Databases DB2 Oracle, DB2 MSSQL1

Servers Mainframe1 UNIX1, Mainframe1 MS222, MS223, MS224

Storage 33801, 33802 SUN101 MSS1, MSS2

Network (LAN, WAN) SLAN1 and SLAN2 SLAN2 and SLAN3 LAN2, LAN3

Procedures and tools Data Center M1 Data Center M1, M2 Data Center M3

People J1, J2 M1, J1 K1, K2

Facilities and physical location

DC1 DC1 DC1

Known vulnerabilities Flood Flood Flood, servers

Estimated cost of outage , per hour

$200K* $150K* $20K*

Business Impact High High Medium

Ranking 1 2 3

Step 1:Collect Info

For Prioritization

Component effect on business processes

Application / Component

Business processes affected

Priority Notes

Patient Records (PR1) PR, Radiology 1

EPIC1 PR, Radiology 2

MSE1 All 4

Mainframe1 All 3

33801 All 5

UNIX1 EPIC1 9 Backup server in place

LAN1 All 10 Backup LAN2 in place

DC1 All 6 DC2 is being planned

Step 1:Collect Info

For Prioritization

Define Vulnerabilities

NATURE Impact Likelihood ranking

PEOPLE Impact Likelihood ranking

EQUIP Impact Likelihood ranking

Fire High Low Human error….

Medium High Applications

High Low

Weather, severe storms

High High Malicious….

Medium Low Servers High Low

Earthquake

High Low Procedure….

Medium High Storage High Low

Water/flood

High High Network…..

High Low

Only Data Centers

Step 2:Vulnerabilities, Risk Assessment, Scope

Define BC target

Business Process

Current Recovery Time:

Success rate %

Budget spent to achieve this

Cost of outage / hour (total $cost avoided)

Target desired Recover Time

Desired success rate %

Projected cost avoidance:

Projected budget

Patient Records

4 hours ??? $100K $200K 30 min 100% ??? ???

Radiology 8 hours ??? $40K $150K 15 min 100% ??? ???

E-mail 30 min good $10K $20K 15 min 100% ??? ???

Step 3: Define BC targetsbased on Scope

Business Process Segmentation

Business Continuity Tier

Availability Disaster Recovery

Recovery Time Objective

Data Currency (Recovery Point objective)

Disaster Recovery performance degradation objective

Acceptable data loss

Sample Business Process

Notes

Continuous Availability

99.99% 1 hour 5 seconds 50% .1% Patient Records, Radiology

(Cost can be listed here)

Rapid Data Recovery

99% 8 hours 1 hour 50% 1% Email, Pharmacy

“

Backup / Restore

98% 36 hours 24 hours 50% 1% Billing “

Step 4:Solution option

design andevaluation

Diagram of primary data center- before

IBMCurrent core

applic.

Various application servers, spread across multiple business lines

Wintel –servers and

growing

LAN/WAN

ATM, Ethernet

Existing storage

1.5 TB, .5 TB, 2 TB, 2 TB, 4 TB, 3 TB, 24 TB ..Various softwares including Veritas Volume Manager, TSM, LTO,

IBM Tape Library, DLT, ArcServe, etc.

MS Exchange, Oracle, web servers, Etc.

…….

Step 4: Solution option design and evaluation

Solution with secondary data center and remote replication

IBM servers

pSeriesEPIC

1.5 TB, .5 TB, 2 TB, 2 TB, 4 TB, 3 TB, 24 TB ..Consolidation: Veritas Volume Manager, TSM, LTO,

IBM Tape Library, etc.

VMWARE for MS Exchange, Oracle, web servers, Etc.

…….

Storage for MS Exchange, Oracle, web servers, etc.

SANSAN Volume Controller(s)

HP Alpha ES47, HP/UX, IDX, PACS, interface engines, Cardiology, CCA lab, pharmacy, radiology, etc

ATM, Ethernet

LAN/WAN

pSeriesEPIC

1.5 TB, .5 TB, 2 TB, 2 TB, 4 TB, 3 TB, 24 TB ..Consolidation: Veritas Volume Manager, TSM,

LTO, IBM Tape Library, etc.

VMWARE for MS Exchange, Oracle, web servers, Etc.

Consolidated storage for MS Exchange, Oracle, web servers, etc.

SAN

SAN Volume Controller(s)

Consolidation: HP Alpha ES47, HP/UX, IDX, PACS, interface engines, Cardiology, CCA lab, pharmacy, radiology, etc

ATM, Ethernet

LAN/WAN

Secondary data centerPrimary data center

Step 5:Recommended

solutionsand products

Suggested Key Performance Indicators for customer IT Business Continuity

Title of KPI: Definition: Measurement: Target:

Backup window Duration daily planned application outage= 20 min

Total time in minutes appl must be quiesced

Reduce to from 20 min to 1 min by 1Q2007

Time to recover (i.e. time to switch sites)

Time to switch sites and restore service

Total time in minutes: outage => users online

Reduce to 30 minutes by YE2007

Testing frequency (preparedness)

Frequency per month of end to end BC test

Number of times/mo for BC test

Improve from 2x/yr to 1x per month

Average system response time

Application response time at bedside

Average and peak response time in sec

In DR mode, maintain no more than 40% increase in resp. time

Average problem resolution time

Average time identify, resolve applic. problem

Average time in hours from initial report

Reduce average time to 2 hours by YE2007

Bandwidth costs Monthly cost of bandwidth to DR site

Dollars/month of expense

Keep % annual expense growth < 20%

How much data is being replicated

Amount of data being replicated by PR,Radio

Total production TB allocated to PR, Radio.

Maintain at 20% of total production TB

Percentage growth rate data

Annual growth % data production PR, Radio.

Quarter to quarter data allocation report

% growth = 10% less than patient growth %

Pro

cedu

ral

Fin

anci

alStep 6:

Recommended Strategy

and Roadmap

18 month IT Business Continuity Implementation phases

Implement BC Tier 4 – Standardize use of disk to disk and Point in Time disk copy

Implement BC Tier 5Standardize DB / Application Mirroring methods

Implement BC Tier 6 – Standardize use of async replic with storage virtualization

Implement BC Tier 7 – Standardize use of Continuous Availability automated failover

Implement BC Tier 3 – Consolidate and standardize Backup/Restore methods. Implement tape Vault, Server / Storage Virtualization / Mgmt tools

Implement 2nd

Site (Tier 1, 2)

Production Backup/Restore BC Tier 1, 2 Foundation:

SAN and server consolidation

Infrastructure Simplification

Backup/Restore BC Tier 1, 2 Foundation:

SAN and server consolidation

Infrastructure Simplification

Backup /Restore

RapidData

Recovery

ContinuousAvailability

Phase 1

Phase 2

Phase 3

Phase 4

Step 6: Recommended Strategy and Roadmap