l b d d A dA utomating data dependability - Courses · l b d d A dA utomating data dependability...

A d d d b lAutomating data dependability

Dr. Kimberly KeetonISM 158: Business Information StrategygyApril 20, [email protected]

© 2006 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice

Problems happen…pp

High Natural disaster — fire, flood, adverse weather

nt

Security breach — hackerMan made disaster — terrorism, malicious damage

Denial of service attack

Hardware failure

per i

ncid

en Virus attackUser/administrator error Software failure

Power failure

Impa

ct p

Network failure

Application failure

Power failure

LowPlanned downtime

2 20 April 2010

LowHighLow Relative frequency

Impact of problemsp p• Data unavailability is expensive – downtime costs/hr:

− Brokerage operations $6,450,000g− Credit card authorization $2,600,000− Ebay (1 outage 22 hours) $ 225,000

Amazon com $ 180 000− Amazon.com $ 180,000− Package shipping services $ 150,000− Home shopping channel $ 113,000− Catalog sales center $ 90,000− Airline reservation center $ 89,000− Cellular service activation $ 41 000Cellular service activation $ 41,000Source: InternetWeek 4/3/2000 + Fibre Channel: A Comprehensive Introduction, R. Kembel 2000, p.8. ”...based

on a survey done by Contingency Planning Research.“

• Data loss is expensive 100 MB $1M ll ll

3 20 April 2010

• Data loss is expensive – 100 MB == $1M (source: Gallup poll)

Impact of data unavailabilityp y

Exponential onsRevenue:

Direct loss, compensatory increase

$ Bi

lli

Direct loss, compensatory payment, lost future revenues, billing losses and investment losses

Damaged reputation

Financial performance

Productivity/ employees

Productivity:Number employee x impacted x hours out = ? $

Impa

ct

reputation

$ M

illio

ns

Constant increase

Ti

Direct financial/ customer

Damaged reputation:Customers, competitors gain advantage, suppliers, financial markets, business

Minutes DaysTimepartners

Financial performance:Revenue recognition, cash flow credit rating stock

Downtime

5 20 April 2010

Indirect impact of downtime can be far more severe and unpredictable

flow, credit rating, stock price, regulatory fines

Anatomy of a failureRecovery metricsRecovery metrics

updates

Recovery Time(duration of outage)

updates

normal operation operation

crash!

Recovery Point

operation continues

(e.g., at 2nd site)

Recovery Point(go back to when?)

timeRPO = max allowedi t ti

100%

availability

recovery-point time (data loss) RTO = max

outage timeallowed

6 20 April 2010

time0%outageallowed

Not all applications are created equalNot all applications are created equalRequirements Category

Business Process Typical Data Form

RTO: min-1 hrRPO: < 30 min

RTO: < 8 hrRPO: < 4 hr

RTO, RPO: < 24 hr

Business ProcessinggData Base

Decision Support

Collaborative Email

IT InfrastructureFile

Web Infrastructure

Scientific/EngineeringVaries by

applicationApplication Development

10 20 April 2010

other

Data protection techniquesData protection techniques

local copy (mirror)

nearby remote copy

primarycopy

partial copy (snapshot)

tape-librarybackup copy

distantremote

continuous data (CDP)

vault: offsite

11 20 April 2010

remote copy

protection (CDP) tape storage

Determining the right solutionOutlay costs Penalty costsOverall costs =

outlays + penalties

Minimize ll

Cost

Spend more, Spend less,

overall costs

Spend more, lose less

p ,lose more

Reco er time Reco er point

12 20 April 2010

Recovery time, Recovery point

Today’s approachy pp• Best practices

− Manual, ad hoc design techniquesT l f d d l− Templates for standard solutions

• Insufficient tools for Insufficient tools for − Evaluating dependability provided by a candidate solution− Examining wide range of candidate designs

Translating b siness req irements into design recommendations− Translating business requirements into design recommendations

• Result− Current solutions are likely conservative (and hence too costly)− Only qualitative understanding of solution dependability

13 20 April 2010

Data dependability overviewData dependability overviewCharacterize goals and failure consequences in financial terms

Formulate data dependability questions as optimization problems

Problem inputs1. Business objectives

penalty rates2. Threats

dependability assessments

failure types

dependability designs

Optimization engine

Modelsrecovery

schedules• workloads• disk arrays• mirroring• tape backup Model cost and dependability

15 20 April 2010

• spare resources Model cost and dependability properties of common data protection

and recovery techniques

Topics to coverTopics to cover• Problem and motivation• Data protection techniques refresher−Backup−Online point-in-time copies−Continuous data protection

/−Remote replication/mirroring

• Research in automating data dependability• Summary

16 20 April 2010

BackupBackup• What?

R d l d − Read-only copy on separate media, representing recovery point in past

• Why?y− To permit time-travel recovery (e.g., before mistake or virus

infection)− To create archival copyo c ea e a c va copy− To enable cheap disaster recovery from offsite vault

• How?− Full vs. partial backups− File vs. block-level backup− Tape-based vs. disk-based architectures

17 20 April 2010

Tape based vs. disk based architectures

Full vs partial backupsFull vs. partial backups• Full backup

− Copy everything backup− Copy everything backup− Restoration is straightforward− Large resource requirements (server, network, tapes)

• Incremental backup• Incremental backup− Cumulative incremental: copy data that’s changed since last full

backup− Differential incremental: copy data that’s changed since last

f ll/ l b kpy g

full/incremental backup− Massive reduction in data moved/copied− Requires more complicated restoration

F ll b k h l i i l h diff i l • Full backup, then cumulative incremental, then one or more differential incrementals

• Most backup schedules employ a combination− Full backup over weekend daily cumulative incrementals during

18 20 April 2010

− Full backup over weekend, daily cumulative incrementals during week

What gets backed upWhat gets backed up• File-level backup

A h f l f l b b k d − Any change to file causes entire file to be backed up− Open files often require special handling software− Pro: simplifies backup and recoveryp p y− Con: small changes to large files result in large backups

• Block-level backupf− Only back up blocks that change in a file

− Requires additional client-side processing to discover changed blocks

− Pro: reduced size of backup data− Con: client processing increases backup and restore complexity

19 20 April 2010

Tape-based backupTape-based backup• Pros:

L w t th di k b d lt ti− Lower costs than disk-based alternatives− Media can be stored offsite for disaster recovery, compliance− Established process

• Cons:− Performance

• Best performance achieved only through streaming and multiplexingp y g g p g• Single file restore operations may be slow

− Device sharing (complex schedules, potential for resource conflicts)− Reliability of devices and mediay

• Robots (and drives) may suffer mechanical failures• Media must be periodically replaced

− Risk of tape loss, incompatibility

25 20 April 2010

p , p y

Disk-based backupDisk-based backup• Benefits:

− Performance− Performance• Online disk storage means faster backups and restores possible• Aggregate performance possible without tape multiplexing

− Reliabilityy• Inherent RAID protection• No robots, so mechanical failure less likely

• Virtual tape libraries (VTL) − Basic idea: virtual disk system that emulates tape library, drives

and media− Can be used to create tape backup for offsite storage− Benefit: requires only incremental changes to existing environment

26 20 April 2010

Online point-in-time copiesOnline point-in-time copies• What?

Di k b d "i t t " th t t d t t ifi i t i − Disk-based "instant copy" that captures data at a specific point in time

− May be read-only or read-writeWhy?• Why?− Allows for complete backup/restore, with very short outages− Provides basis for zero downtime backup− May permit restoration of individual files (depending on

implementation)− Allows for time travel recovery (may be more limited than backup)

d f d− Provides support for data repurposing• How?

− Full or partial point-in-time copies

27 20 April 2010

p p p

Full point-in-time copiesFull point-in-time copies• Also known as split mirrors and clones

d k l l d f d• Basic idea: keep multiple RAID-1 (mirrored) sets of data− Freeze content of one set by ceasing RAID-1 operations

• Can be done by variety of sources• Can be done by variety of sources− Host-based logical volume manager− Fabric-based virtualizer or logical volume manager− Array-based functions

• Frozen copy is independent of current copy for contentB tt f i l ti t l f f il− Better performance isolation, tolerance for failure

• Resynchronization requires explicit copying of data

28 20 April 2010

Partial (differential) point-in-time copiesPartial (differential) point-in-time copies• Also known as snapshots

d k h h• Basic idea: track "changes" to the primary copy− “Change" disks + primary disks provide PiT copy

• Can be done by variety of sources: file system disk array• Can be done by variety of sources: file system, disk array• Several approaches, each with performance impact:

− Copy on write (CoW): copy old data before update in placepy ( ) py p p− Redirect on write (RoW): new updates stored in journal− Write anywhere (WA): non-overwrite updates to virtual blocks;

maps for “now” and “snapshots”maps for now and snapshots

• Implications of leveraging main copy:− Less storage consumption than full PiT copies

29 20 April 2010

− Can’t use for disaster recovery

Continuous data protection (CDP)Continuous data protection (CDP)• What?

C ll d h d − Capture all updates as they occur, not just discrete points in time− Store protected copy in secondary location

• Why?Why?− To provide very fine granularities of recovery points

• How?− May be block-, file- or application-based− Very resource-intensive, so period of protection may be limited

34 20 April 2010

Remote replication/mirroringRemote replication/mirroring• What?

l bl k l k d ff−File or block replication across network to different location

−Provides up-to-date copy at a different location p py• Why?−To permit application failover and failback after site

di tdisaster−To migrate data or distribute content between locations−To permit remote backupTo permit remote backup

• How?−Synchronous or asynchronous protocols

36 20 April 2010

y y p

Synchronous remote replicationSynchronous remote replication• Basic idea−Each update is applied at both local and remote site

before acknowledgment to local hostProvides completely up to date replica−Provides completely up-to-date replica

• IssuesInterconnect must be provisioned to handle write bursts − Interconnect must be provisioned to handle write bursts

−Speed of light latency means only appropriate for regional distancesg

−Must resynchronize mirrors after network partition

37 20 April 2010

Asynchronous remote replicationAsynchronous remote replication• Basic idea

A k l d h f l l d d− Acknowledge write to host after local copy updated− Batch together writes within an interval− Apply update batch atomically at remote sitepp y p y

• Implications− Remote copy lags local copy slightly

f f− Overwrites will reduce amount of data to be transferred− Interconnect bandwidth provisioned based on desired RPO− Not limited geographicallyg g p y− Must resynchronize mirrors after network partition

38 20 April 2010

Topics to coverTopics to cover• Problem and motivation• Data protection techniques refresher• Research in automating data dependabilityg p y−Dependability evaluation−Dependability design−Recovery scheduling

• Summary

40 20 April 2010

Data dependability overviewData dependability overviewCharacterize goals and failure consequences in financial terms

Problem inputs1. Business objectives

penalty rates2. Threats

Formulate data dependability questions as optimization problems

failure types

dependability assessments

Models

Optimization engine

dependability designs

• workloads• disk arrays• mirroring• tape backup

recoveryschedules

Model cost and dependability

41 20 April 2010

• spare resources Model cost and dependability properties of common data protection

and recovery techniques

Managing data by business value: penalty ratespenalty rates

failureStringent requirements

$/hour

sec min hr day weekweek day hr min sec

Relaxed requirements

• Recovery time objective (RTO):

RTORPOyy

Data outage penalty rateData loss penalty rate

y j ( )− How long before the system is back up?

• Recovery point objective (RPO): h d h d d− How much recent data can the system discard?

• Penalty rate model− Data loss penalty rate ($/hour)

42 20 April 2010

Data loss penalty rate ($/hour)− Data outage penalty rate ($/hour)

Modeling data protection techniques[Keeton and Merchant DSN2004][Keeton and Merchant – DSN2004]

• Primary copy of data is protected by one or more Level 0: primary copy protected by one or more secondary point-in-time copies

• Techniques create, retain and

Level 0: primary copy

Level 1: d

diskarray

qpropagate copies− Parameters for frequency of

copies, time to create copies,

secondary copy

near online disk

remote mirror

p pretention

• Model copies as hierarchyIncreasing levels:L l i

… near-online disk (snapshots)

Increasing levels:− larger retention capacity − longer recovery latencies− less frequent PiT copies

Level i

offline tape (vault)

near-line tape

43 20 April 2010

less frequent PiT copiesoffline tape (vault)

Dependability assessment case studyDependability assessment case study• Research workgroup file server application

f l l−Business requirements: financial penalty rates• Downtime penalty rate: $50,000 / hour• Recent data loss penalty rate: $50,000 / hourp y , /

−Workload characteristics: light to moderate usage• Data capacity: 1.4 TB• Average access rate: 1028 KB / sec• Average access rate: 1028 KB / sec• Average update rate: 799 KB / sec• Peak:average update ratio: 10X

D l d 317 KB / • Daily unique update rate: 317 KB / sec

• Questions to answer:Does existing solution meet goals?

44 20 April 2010

−Does existing solution meet goals?−What if we changed the configuration to . . . ?

Case study: backup and vaultingCase study: backup and vaultingPrimary building/site

Shared spare site

Secondary site

Tape vaultPrimary array

remotevaulting

Storage-areanetwork

Host Host

Tape lib

Tape lib

primary copy Disk

arraytape

array vaultingnetwork

• Baseline: split mirrors every 12 hours, weekly full backup (48 hr backup window), monthly remote vaulting

split mirror backup

) y g• Weekly vault: baseline, except weekly remote vaulting• Weekly vault, F+I: weekly vault, plus daily incremental backups (12 hr backup

window)• Weekly vault daily F: weekly vault except daily full backups (12 hr backup

45 20 April 2010

Weekly vault, daily F: weekly vault, except daily full backups (12 hr backup window)

Case study: remote mirroringCase study: remote mirroringPrimary building/site

Secondary sitePrimary

arrayStorage-area

network

Host Host

Tape lib

primary copy Disk

array

remote mirror

• AsyncB mirror, 1 link: asynchronous remote mirroring with y , y g1min windows, 1 network link

• AsyncB mirror, 10 links: asynchronous remote mirroring with 1min windows, 10 links

46 20 April 2010

w w dows, 0 s

Recovery behaviorRecovery behavior• Primary array failure • Site disaster

5

6

7

5

6

7

AsyncB mirror, 1 link

AsyncB mirror, 10 links

2

3

4

2

3

4

Weekly vaultWeekly vault, F+I

Weekly vault, daily F

0

1

-1500 -1000 -500 0 500

0

1

-250 -200 -150 -100 -50 0 50

Weekly vault

Baseline

Recovery point (hr) Recovery time (hr)

Recovery point (hr) Recovery time (hr)

47 20 April 2010

Financial ramificationsFinancial ramifications

18

20

$) Penalty costs (loss)

12

14

16

mill

ions

$ Penalty costs (outage)Outlay costs

8

10

12

cost

s (m

2

4

6

Ove

rall

0

2

Baseline Weeklyvault

Weeklyvault F+I

Weeklyvault daily

AsyncBmirror 1

AsyncBmirror 10

48 20 April 2010

vault vault, F+I vault, dailyF

mirror, 1link

mirror, 10links

Storage configuration

Automated dependability designs [Keeton, et al. – FAST2004], [Gaonkar, et al. – DSN2006]

alty

rate

ur

)

Central BankWeb

server

(500,500K)(50M,5M)

utag

e pe

na($

/hou

Consumer banking

Student Company

(500,500) (500K,500)(50M,50K)

Ou

Data loss penalty rate($/hour)

Studentaccounts

Company Docs

($/hour)• What is best storage design to meet business requirements?• Experimental design

− Penalty rates for five different industry segments− Same workload ([cello2002])− Annualized outlay costs one site disaster per year

April 20, 2010 49

− Annualized outlay costs, one site disaster per year− Solver determines best design

Automated dependability designs

alty

rate

ur

)

Central BankWeb

server

(500,500K)(50M,5M)

utag

e pe

na($

/hou

Consumer banking

Student Company

(500,500) (500K,500)(50M,50K)

Ou

Data loss penalty rate($/hour)

Studentaccounts

Company Docs

Industry segment Automated design choice

Student accts Backup + 12-hr win + no spares

($/hour)

Company docs Async + recovery + no sparesWeb server Batched async + failoverC b ki S + f il

April 20, 2010 51

Consumer banking Sync + failoverCentral bank Sync + failover

Design choice cost breakdown

70%80%90%

100%

rall

cost

s $301K $426K $506K $562K $603K

30%40%50%60%70%

on to

ove

r

0%10%20%30%

Student Company Web server Consumer Central bankCon

trib

utio

Studentaccounts

Companydocuments

Web server Consumerbanking

Central bank

Industry segment

C

Primary Array Outlays Mirroring Outlays Backup OutlaysSpare Resources Outlays Data Unavailability Penalties Data Loss Penalties

• Low penalty rates (student accounts, company docs): − Non-trivial penalty component ; outlays for more aggressive solutions

Spare Resources Outlays Data Unavailability Penalties Data Loss Penalties

April 20, 2010 52

Non trivial penalty component ; outlays for more aggressive solutions outweigh penalties


70%80%90%

100%

rall

cost

s $301K $426K $506K $562K $603K

30%40%50%60%70%

on to

ove

r

0%10%20%30%


trib

utio

Studentaccounts

Companydocuments


Central bank

Industry segment

C


• Moderate penalty rates (web server, consumer banking): − Mirroring outlays dominate


April 20, 2010 53

Mirroring outlays dominate− These outlays are less than penalties for less aggressive solutions


70%80%90%

100%

rall

cost

s $301K $426K $506K $562K $603K

30%40%50%60%70%

on to

ove

r

0%10%20%30%


trib

utio

Studentaccounts

Companydocuments


Central bank

Industry segment

C


• High penalty rates (central bank):− Non-trivial data outage penalty for aggressive solution


April 20, 2010 54

Non trivial data outage penalty for aggressive solution

Design space exploration50 000 000

5 000 000 /hr)

5 000 000

500 000

y ra

te ($

/

Batched async

mirror Sync

mirror

50 000

e pe

naltymirror

5 000

500 ta o

utag

e

Backup

Failover

Reconstruct500

Dat

00 0

00

00 0

00

00 0

00

50 0

00

5 00

0

500

Async mirror

April 20, 2010 55

50 0

0

5 00505

Data loss penalty rate ($/hr)

Design overall (outlay + penalty) costs

1200

1400

1 200

1 400Web serverC t l b k

600

800

10001 000

600

800 Overall cost Student accounts

Company docs

Central bank

158

1580

5800

0

200

400400

200

Data loss

(k$)

Consumer bank

1580

1580

0015

8000

000

000

11580

15800

158000

1580000

15800000

Data loss

penalty rate ($/hr)

April 20, 2010 56

1580

0 158580 0

Data outage

penalty rate ($/hr)

Recovering data systems [Keeton, et al. - EuroSys2006][ y ]

site2

array3 array4

site1

array2array1

Restore from mirror

failover

failback

array3 array4array2array1

C DW SB

C DW SB

CB W D S

acku

p

link1

l k

svr6servers

svr2 svr4svr3 svr5svr1

C W D SBre fr

om b

a

link2vault1library1

B C W D S B C W D S

Res

tor

= primary copy = application = secondary copy

Restorefrom vault

58 20 April 2010

p y py pp y py

• Goal: choose alternatives and recovery schedule that minimize total penalties over all workloads

Modeling recovery time: recovery schedulingscheduling• How to recover dependable storage system post-disaster?

Which copy to recover each failed application?− Which copy to recover each failed application?− How to schedule resources among competing demands?

• Recovery graphs describe potential recovery strategiesy g p p y g

Reprovision F ilb k A & A b kR t t R t t L lReprovision Failback App &mirroring

App, backup& mirroring

Restart mirroring

Restart backup

Localapp

Remote appFailover

• Apply optimization to determine best solution− Choose recovery strategies and recovery schedule that minimize total

64 20 April 2010

Choose recovery strategies and recovery schedule that minimize total penalties over all workloads

Describing the problem formallyg p y• Objective

− Common currency: penalty costs for downtime, loss of recent y p y ,data, and vulnerability to subsequent failures

− Minimize total penalty costs over all workloads

• Decisions • Decisions − Select a recovery path (alternative)− Schedule for jobs in selected paths

• Constraints− Exactly one recovery path is chosen for each workload

J b ti ti fi h d t i t− Job execution satisfies recovery graph precedence constraints− Job consumes resources while it is being executed− Job scheduling satisfies resource capacity constraints

66 20 April 2010

Optimal recovery schedulep y

Commercial Bankk

Failover mirror

Recovery pathScheduleWorkload

Consumer BankWeb server

Restore mirrorRestore backup

Outage (time) Penalty costsCommercial Bank

Consumer BankWeb Server

Outage (time)$5.0 M$1.6 M

$12.2 M

Penalty costs

Remote server 100% 0%

Network 20%21% 71% 81% 50%

100% 0%

7% 7%Tape library 3% 8%5%

15% 8%Local array 1 Failed 12%12%

68 20 April 2010

Lessons learnedLessons learned• Financial business requirements allow optimization

l h b l d• Formal optimization techniques can be applied− Mixed Integer Program (MIP) formulations provide optimal solution

for small problemsp− Heuristics needed to solve larger problems− MIP useful for understanding problem structure

I t d t ll ti i h ll i• Input data collection is challenging− Use storage management tools to automatically discover− Build templates of common scenariosp

• Storage is important start, but users ultimately care about application and business process dependability

69 20 April 2010

SummarySummary• Data unavailability and loss is very expensive

M t h i f t ti d t• Many techniques for protecting data− Backup, online point-in-time copies, remote replication,

etc.• Understanding right techniques to use requires

holistic approachh d d d b l• Our approach to automating data dependability

− Characterize business requirements in financial termsM d l d d bilit f d t t ti d − Model dependability of data protection and recovery techniques

− Formulate scheduling, design questions as optimization bl

71 20 April 2010

problems

For more informationFor more information…•Background on data protection techniques:

N A lSNIA tutorials−Michael Fishman, “Introduction to Data Protection: Backup

t T Di k d B d”to Tape, Disk and Beyond”− Jason Iehl and Andreas Schwegman, “Trends in Data

Protection and Restoration Technologies”g

• Available from Available from http://www.snia.org/education/tutorials

72 20 April 2010

For more informationFor more information…• Research in automating data dependability:

“D i i f di ” K K C S D B J Ch − “Designing for disasters,” K. Keeton, C. Santos, D. Beyer, J. Chase and J. Wilkes, Proc. 3rd Conference on File and Storage Technologies (FAST), March 2004.

− “A framework for evaluating storage system dependability ” K Keeton A framework for evaluating storage system dependability, K. Keeton and A. Merchant, Proc. Intl. Symposium on Dependable Systems and Networks (DSN), June 2004.

− “On the road to recovery: restoring data after disasters,” K. Keeton, D B E B A M h t C S t d A Zh P f D. Beyer, E. Brau, A. Merchant, C. Santos and A. Zhang. Proc. of ACM European Systems Conference (EuroSys), Leuven, Belgium, April 2006.

− “Designing dependable storage solutions for shared application es g g depe dab e s o age so u o s o s a ed app ca o environments,” S. Gaonkar, K. Keeton, A. Merchant, and W. Sanders. Proc. of Intl. Conf. on Dependable Systems and Networks (DSN) , Philadelphia, PA, June 2006.

A il bl f htt // h l h /SSP/ /

73 20 April 2010

• Available from http://www.hpl.hp.com/SSP/papers/

l b d d A dA utomating data dependability - Courses · l b d d A dA utomating data dependability...

Documents

Transcript of l b d d A dA utomating data dependability - Courses · l b d d A dA utomating data dependability...