ReViveI/O: Efficient Handling of I/O in Highly...
151
ReViveI/O: Efficient Handling of I/O in Highly-Available Rollback-Recovery Servers Jun Nakano ✢ , Pablo Montesinos, Kourosh Gharachorloo ❆ , Josep Torrellas University of Illinois at Urbana-Champaign ✢ IBM ❆ Google
Transcript of ReViveI/O: Efficient Handling of I/O in Highly...
ReViveI/O:
Efficient Handling of I/O in Highly-Available Rollback-Recovery
ServersJun Nakano✢, Pablo Montesinos, Kourosh Gharachorloo❆, Josep Torrellas
University of Illinois at Urbana-Champaign✢IBM❆Google
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Building Fault-Tolerant Systems
2
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Building Fault-Tolerant Systems
2
RedundantSelf-Checking
HW
High OverheadSmall/Null MTTR
HP NonstopIBM S/390
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Building Fault-Tolerant Systems
2
RedundantSelf-Checking
HW
High OverheadSmall/Null MTTR
HP NonstopIBM S/390
Plain HW +SW-based
Checkpointing
High OverheadSignificant MTTR
KeyKOSFT Mach
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Building Fault-Tolerant Systems
2
RedundantSelf-Checking
HW
High OverheadSmall/Null MTTR
HP NonstopIBM S/390
Plain HW +SW-based
Checkpointing
High OverheadSignificant MTTR
KeyKOSFT Mach
HW-based High-frequency Checkpointing
Low overheadSmall MTTR
ReVive, SafetyNet
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
A glimpse of ReVive (ISCA 2002)
3
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
A glimpse of ReVive (ISCA 2002)
• Goal: recover the memory state of a shared-memory machine in < 1s
3
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
A glimpse of ReVive (ISCA 2002)
• Goal: recover the memory state of a shared-memory machine in < 1s
• HW-assisted very frequent checkpoint (20ms < T < 100ms)
• During checkpoints:
• Write-back dirty data from caches
• Main memory is the checkpoint state
• Between checkpoints:
• HW logs overwritten data in memory when is modified for the first time
3
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
A glimpse of ReVive (ISCA 2002)
• Goal: recover the memory state of a shared-memory machine in < 1s
• HW-assisted very frequent checkpoint (20ms < T < 100ms)
• During checkpoints:
• Write-back dirty data from caches
• Main memory is the checkpoint state
• Between checkpoints:
• HW logs overwritten data in memory when is modified for the first time
• Entire main memory protected by distributed parity
• Like RAID-5, but in memory
• Can tolerate the loss of a node
3
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
HW-based High-frequency Checkpointing
Low overheadSmall MTTR
ReVive,SafetyNet
Building Fault-Tolerant Systems
4
RedundantSelf-Checking
HW
High OverheadSmall/Null MTTR
HP NonstopIBM S/390
Plain HW +SW-based
Checkpointing
High OverheadSignificant MTTR
KeyKOSFT Mach
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
RedundantSelf-Checking
HW
Expensive
Building Fault-Tolerant Systems
5
Plain HW +SW-based
Checkpointing
High OverheadSignificant MTTR
KeyKOSFT Mach
HW-based High-frequency Checkpointing
Low overheadSmall MTTR
ReVive,SafetyNet
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
RedundantSelf-Checking
HW
Expensive
Building Fault-Tolerant Systems
6
Plain HW +SW-based
Checkpointing
Significant MTTR
HW-based High-frequency Checkpointing
Low overheadSmall MTTR
ReVive,SafetyNet
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
RedundantSelf-Checking
HW
Expensive
Building Fault-Tolerant Systems
7
Plain HW +SW-based
Checkpointing
Significant MTTR
HW-based High-frequency Checkpointing
Modest HWNo I/O support
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
RedundantSelf-Checking
HW
Expensive
Building Fault-Tolerant Systems
8
Plain HW +SW-based
Checkpointing
Significant MTTR
HW-based High-frequency Checkpointing
Modest HWNo I/O support
(until today)
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Why is I/O undo/redo hard?
9
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
• Output Commit Problem:
Why is I/O undo/redo hard?
9
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
• Output Commit Problem:
• An output to the external world cannot be rolled back
• st
Why is I/O undo/redo hard?
9
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
• Output Commit Problem:
• An output to the external world cannot be rolled back
Why is I/O undo/redo hard?
10
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
• Output Commit Problem:
• An output to the external world cannot be rolled back
Why is I/O undo/redo hard?
10
Server
Disk
Network
Checkpoint n Checkpoint n+1
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
• Output Commit Problem:
• An output to the external world cannot be rolled back
Why is I/O undo/redo hard?
10
Server
Disk
Network
Checkpoint n Checkpoint n+1
write
send
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
• Output Commit Problem:
• An output to the external world cannot be rolled back
Why is I/O undo/redo hard?
10
Server
Disk
Network
Checkpoint n Checkpoint n+1
write
send
X
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
• Output Commit Problem:
• An output to the external world cannot be rolled back:
• How do you undo the I/O? fjkshalkjfhlkjsdfhkldjhklsdhkljhkhlkhklhk
Why is I/O undo/redo hard?
11
Server
Disk
Network
Checkpoint n Checkpoint n+1
write
send
X
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Contribution: ReViveI/O
12
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Contribution: ReViveI/O
• Support I/O undo/redo in rollback-recovery SMP servers
12
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Contribution: ReViveI/O
• Support I/O undo/redo in rollback-recovery SMP servers
• Targeted to throughput-oriented workloads
12
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Contribution: ReViveI/O
• Support I/O undo/redo in rollback-recovery SMP servers
• Targeted to throughput-oriented workloads
• Disk and network only
12
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Contribution: ReViveI/O
• Support I/O undo/redo in rollback-recovery SMP servers
• Targeted to throughput-oriented workloads
• Disk and network only
• Transparent to OS and applications
12
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Contribution: ReViveI/O
• Support I/O undo/redo in rollback-recovery SMP servers
• Targeted to throughput-oriented workloads
• Disk and network only
• Transparent to OS and applications
• Low overhead during fault-free operation (<1% over ReVive)
12
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Contribution: ReViveI/O
• Support I/O undo/redo in rollback-recovery SMP servers
• Targeted to throughput-oriented workloads
• Disk and network only
• Transparent to OS and applications
• Low overhead during fault-free operation (<1% over ReVive)
• Fast recovery (< 1s even if processor lost)
12
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Contribution: ReViveI/O
• Support I/O undo/redo in rollback-recovery SMP servers
• Targeted to throughput-oriented workloads
• Disk and network only
• Transparent to OS and applications
• Low overhead during fault-free operation (<1% over ReVive)
• Fast recovery (< 1s even if processor lost)
• Small impact on hardware cost
12
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Approach: Pseudo-Device Driver
13
• Masubuchi et al, FTCS-97
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Approach: Pseudo-Device Driver
13
Device
• Masubuchi et al, FTCS-97
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Approach: Pseudo-Device Driver
13
Device
Device Driver
• Masubuchi et al, FTCS-97
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Approach: Pseudo-Device Driver
13
Device
Device Driver
Kernel
• Masubuchi et al, FTCS-97
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Approach: Pseudo-Device Driver
13
Device
Device Driver
Kernel
Pseudo Device-Driver(PDD)
• Masubuchi et al, FTCS-97
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Approach: Pseudo-Device Driver
• Delays outputs until next checkpoint
• Commits outputs after next checkpoint
13
Device
Device Driver
Kernel
Pseudo Device-Driver(PDD)
• Masubuchi et al, FTCS-97
• No need to modify OS nor applications
• Network and Disk PDDs
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Fault Model
14
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Fault Model
14
Fault!
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Fault Model
14
MemoryRecoverable by ReVive
(MR)
MR=Transients or 1 permanent with at most 1 node loss
Fault!
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Fault Model
14
ReVive restores memory from previous checkpoint
MemoryRecoverable by ReVive
(MR)
PDD recovers I/O transparently
Yes
MR=Transients or 1 permanent with at most 1 node loss
Fault!
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Fault Model
14
ReVive restores memory from previous checkpoint
MemoryRecoverable by ReVive
(MR)
PDD recovers I/O transparently
Yes No
MR=Transients or 1 permanent with at most 1 node loss
Fault!
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Fault Model
14
ReVive restores memory from previous checkpoint
MemoryRecoverable by ReVive
(MR)
Fix HW and Reboot
PDD recovers I/O transparently
Yes No
MR=Transients or 1 permanent with at most 1 node loss
Fault!
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Fault Model
14
ReVive restores memory from previous checkpoint
MemoryRecoverable by ReVive
(MR)
PDD makes the I/O state consistent
Fix HW and Reboot
PDD recovers I/O transparently
Yes No
MR=Transients or 1 permanent with at most 1 node loss
Fault!
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Fault Model
14
ReVive restores memory from previous checkpoint
MemoryRecoverable by ReVive
(MR)
PDD makes the I/O state consistent
Conventional recovery by Application (e.g. DB)
Fix HW and Reboot
PDD recovers I/O transparently
Yes No
MR=Transients or 1 permanent with at most 1 node loss
Fault!
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
• ReViveI/O leverages the properties of TCP to support network undo/redo
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
PKTi
• ReViveI/O leverages the properties of TCP to support network undo/redo
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
PKTi
• ReViveI/O leverages the properties of TCP to support network undo/redo
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
PKTi
ACKi
• ReViveI/O leverages the properties of TCP to support network undo/redo
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
PKTi
ACKi
• ReViveI/O leverages the properties of TCP to support network undo/redo
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
x
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
• ReViveI/O leverages the properties of TCP to support network undo/redo
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
x
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
• ReViveI/O leverages the properties of TCP to support network undo/redo
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
ACKi
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
ACKi
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
ACKi
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
ACKi
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
ACKi
PKTi+1
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
ACKi
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
ACKi
PKTi+1
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
ACKi
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
ACKi
PKTi+1
ACKi+1
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
ACKi
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
ACKi
PKTi+1
ACKi+1
ACKi+1
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
ACKi
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
ACKi
PKTi+1
ACKi+1
ACKi+1x
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
ACKi
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
ACKi
PKTi+1
ACKi+1
x
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
ACKi
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
ACKi
PKTi+1
ACKi+1
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
ACKi
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
• TCP eliminates duplicate packages
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
ACKi
PKTi+1
ACKi+1
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
15
ACKi
• ReViveI/O leverages the properties of TCP to support network undo/redo
• TCP avoids saving packages for inputs: the sender timeouts and retransmits
• TCP eliminates duplicate packages
Network PDD
Kernel, App
Client
CKPTi CKPTi+1 CKPTi+2 CKPTi+3 CKPTi+4 CKPTi+5
Time
PKTi
ACKi
PKTi+1
ACKi+1
ACKi+1
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD Schemes
16
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD Schemes
16
Scheme On I/O write issue After next checkpoint
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD Schemes
16
Scheme On I/O write issue After next checkpoint
Stall Stall the write Write data to disk
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD Schemes
16
Scheme On I/O write issue After next checkpoint
Stall Stall the write Write data to disk
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD Schemes
16
Scheme On I/O write issue After next checkpoint
Stall Stall the write Write data to disk
LoggingCopy old data elsewhere in disk
Write data in placeDelete pointer to old data
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD Schemes
16
Scheme On I/O write issue After next checkpoint
Stall Stall the write Write data to disk
LoggingCopy old data elsewhere in disk
Write data in placeDelete pointer to old data
Buffering Write data to disk buffer and mem Copy data from mem to disk
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD Schemes
16
Scheme On I/O write issue After next checkpoint
Stall Stall the write Write data to disk
LoggingCopy old data elsewhere in disk
Write data in placeDelete pointer to old data
Buffering Write data to disk buffer and mem Copy data from mem to disk
RenamingWrite data to renamed disk location
Save old logical → physical mapping Delete old mapping
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD Schemes
17
Scheme On I/O write issue After next checkpoint
Stall Stall the write Write data to disk
LoggingCopy old data elsewhere in disk
Write data in placeDelete pointer to old data
Buffering Write data to disk buffer and mem Copy data from mem to disk
RenamingWrite data to renamed disk location
Save old logical → physical mapping Delete old mapping
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Memory Buffer
Disk Buffer
Target Disk
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
update request
Memory Buffer
Disk Buffer
Target Disk
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
update request
Memory Buffer
Disk Buffer
Target Disk
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
update request
Memory Buffer
Disk Buffer
Target Disk
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
update request
Memory Buffer
Disk Buffer
Target Disk
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
update request
Memory Buffer
Disk Buffer
Target Disk
x
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Memory Buffer
Disk Buffer
Target Disk
x
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Memory Buffer
Disk Buffer
update request
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Memory Buffer
Disk Buffer
update request
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Memory Buffer
Disk Buffer
update request
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Memory Buffer
Disk Buffer
update request
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Memory Buffer
Disk Buffer
update request
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
response
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Memory Buffer
Disk Buffer
update request
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
response
x
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Memory Buffer
Disk Buffer
update request
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
x
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Memory Buffer
Disk Buffer
update request
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Memory Buffer
Disk Buffer
update request
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Memory Buffer
Disk Buffer
update request response
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Memory Buffer
Disk Buffer
update request response
x
Non MR Fault
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Memory Buffer
Disk Buffer
update request
x
Non MR Fault
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Disk Buffer
Non MR Fault
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD
18
Target Disk
MR Fault
Time
Network PDD
Kernel, App
Client
CKPTi CKPTi+3 CKPTi+5
Disk PDD
CKPTi+2 CKPTi+4
Disk Buffer
Non MR Fault
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
19
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
19
Useful Work
CKP
Time
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
19
Useful Work
CKP
Time 100ms
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
20
Lost Work
Useful Work
CKP
Time 100ms
X
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
20
Lost Work
Useful Work
CKP
Time 100ms
Detection
80ms
X
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
20
Lost Work
Useful Work
CKP
Time 100ms
Detection
80ms
•Self-Check•Rerouting
50ms
X
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
20
Lost Work
ReVive Recovery
Useful Work
CKP
Time 100ms
Detection
80ms
•Self-Check•Rerouting
50ms
ReconstructLost Log
~100ms
X
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
20
Lost Work
ReVive Recovery
Useful Work
CKP
Time 100ms
Detection
80ms
•Self-Check•Rerouting
50ms
ReconstructLost Log
~100ms
Rollback
~490ms
X
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
20
Lost Work
ReVive Recovery
Machine Unavailable
Degraded Execution
Useful Work
CKP
Time 100ms
Detection
80ms
•Self-Check•Rerouting
50ms
ReconstructLost Data
~seconds
ReconstructLost Log
~100ms
Rollback
~490ms
X
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
21
Lost Work
ReVive Recovery
Machine Unavailable
Degraded Execution
Useful Work
CKP
Time 100ms
Detection
80ms
•Self-Check•Rerouting
50ms
ReconstructLost Data
~seconds
ReconstructLost Log
~100ms
Rollback
~490ms
X
ReVive I/O Recovery
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
21
Lost Work
ReVive Recovery
Machine Unavailable
Degraded Execution
Useful Work
CKP
Time 100ms
Detection
80ms
•Self-Check•Rerouting
50ms
ReconstructLost Data
~seconds
ReconstructLost Log
~100ms
Rollback
~490ms
X
ReVive I/O Recovery
Reset Device(~1ms)
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
21
Lost Work
ReVive Recovery
Machine Unavailable
Degraded Execution
Useful Work
CKP
Time 100ms
Detection
80ms
•Self-Check•Rerouting
50ms
ReconstructLost Data
~seconds
ReconstructLost Log
~100ms
Rollback
~490ms
X
ReVive I/O Recovery
Reset Device(~1ms)
Reset Device Driver(~1ms)
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Recovery: ReVive + ReViveI/O
21
Lost Work
ReVive Recovery
Machine Unavailable
Degraded Execution
Useful Work
CKP
Time 100ms
Detection
80ms
•Self-Check•Rerouting
50ms
ReconstructLost Data
~seconds
ReconstructLost Log
~100ms
Rollback
~490ms
X
ReVive I/O Recovery
Reset Device(~1ms)
Reset Device Driver(~1ms)
Re-issue Outputs in theBackground (<100ms)
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Experiments
22
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Experiments
22
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Experiments
22
Clients
•Dual 1.5 GHz Athlon•Gigabit Ethernet•1 GB RAM
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Experiments
22
Clients
•Dual 1.5 GHz Athlon•Gigabit Ethernet•1 GB RAM
Server
•Dual 1.5 GHz Athlon•Gigabit Ethernet•2 GB RAM
+ ReVive PDD
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Throughput-oriented workload
23
1.11.00.90.80.70.60.50.40.30.20.10
Checkpoint interval (ms)
Norm
aliz
ed T
hro
ughput
(%)
TPC-C on Oracle9i with 30 Clients
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Throughput-oriented workload
23
20 40 80 120 160 200 240
1.11.00.90.80.70.60.50.40.30.20.10
Checkpoint interval (ms)
Norm
aliz
ed T
hro
ughput
(%)
TPC-C on Oracle9i with 30 Clients
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Throughput-oriented workload
23
20 40 80 120 160 200 240
StallReViveI/O
1.11.00.90.80.70.60.50.40.30.20.10
Checkpoint interval (ms)
Norm
aliz
ed T
hro
ughput
(%)
TPC-C on Oracle9i with 30 Clients
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Throughput-oriented workload
• Stalling scheme (simply delaying writes) incurs too much overhead
23
20 40 80 120 160 200 240
StallReViveI/O
1.11.00.90.80.70.60.50.40.30.20.10
Checkpoint interval (ms)
Norm
aliz
ed T
hro
ughput
(%)
TPC-C on Oracle9i with 30 Clients
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Throughput-oriented workload
• Stalling scheme (simply delaying writes) incurs too much overhead
• ReViveI/O’s overhead is only 0.1% for T<120 ms
23
20 40 80 120 160 200 240
StallReViveI/O
1.11.00.90.80.70.60.50.40.30.20.10
Checkpoint interval (ms)
Norm
aliz
ed T
hro
ughput
(%)
TPC-C on Oracle9i with 30 Clients
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Integration with ReVive (for TPC-C)
24
Checkpoint Interval (ms)
Red
uct
ion in T
hro
ughput
(%
)
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
0 20 40 60 80 100 120 140 160 1800
2
4
6
8
10
12
Integration with ReVive (for TPC-C)
24
Checkpoint Interval (ms)
Red
uct
ion in T
hro
ughput
(%
)
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
0 20 40 60 80 100 120 140 160 1800
2
4
6
8
10
12
Integration with ReVive (for TPC-C)
24
ReVive + ReVive I/O
ReVive Only (estimated)
Checkpoint Interval (ms)
Red
uct
ion in T
hro
ughput
(%
)
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
0 20 40 60 80 100 120 140 160 1800
2
4
6
8
10
12
Integration with ReVive (for TPC-C)
• ReVive + ReViveI/O: only 7% throughput reduction for 60 - 120 ms checkpoint intervals
24
ReVive + ReVive I/O
ReVive Only (estimated)
Checkpoint Interval (ms)
Red
uct
ion in T
hro
ughput
(%
)
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
4 hr2 1 day 1 week 10 years1 year631 month
99.9%
99.99%
99.999%
99.9999%
99.99999%
MTBFMR
Availability
Availability
25
1 hr
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
4 hr2 1 day 1 week 10 years1 year631 month
99.9%
99.99%
99.999%
99.9999%
99.99999%
MTBFMR
Availability
Baseline (No Rollback)
Availability
25
1 hr
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
4 hr2 1 day 1 week 10 years1 year631 month
99.9%
99.99%
99.999%
99.9999%
99.99999%
MTBFMR
Availability
Baseline (No Rollback)
Availability
25
1 hr
ReVive + ReViveI/O. MTBFNMR = 1000 x MTBFMR
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
4 hr2 1 day 1 week 10 years1 year631 month
99.9%
99.99%
99.999%
99.9999%
99.99999%
MTBFMR
Availability
Baseline (No Rollback)
Availability
• ReVive I/O beats baseline thanks to its low recovery latency for MR faults
25
1 hr
ReVive + ReViveI/O. MTBFNMR = 1000 x MTBFMR
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Conclusions
26
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Conclusions
• Support for I/O undo/redo in memory-checkpointing systems:
• Transparent to OS and applications
• Fast recovery and negligible overhead during fault-free operation
• Low space overhead (see paper)
26
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Conclusions
• Support for I/O undo/redo in memory-checkpointing systems:
• Transparent to OS and applications
• Fast recovery and negligible overhead during fault-free operation
• Low space overhead (see paper)
• Troughtput-oriented workloads:
• < 1% overhead over ReVive
26
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Conclusions
• Support for I/O undo/redo in memory-checkpointing systems:
• Transparent to OS and applications
• Fast recovery and negligible overhead during fault-free operation
• Low space overhead (see paper)
• Troughtput-oriented workloads:
• < 1% overhead over ReVive
• Latency-bound workloads:
• Tolerable checkpoint interval is application dependent (see paper)
26
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Conclusions
• Support for I/O undo/redo in memory-checkpointing systems:
• Transparent to OS and applications
• Fast recovery and negligible overhead during fault-free operation
• Low space overhead (see paper)
• Troughtput-oriented workloads:
• < 1% overhead over ReVive
• Latency-bound workloads:
• Tolerable checkpoint interval is application dependent (see paper)
26
• ReViveI/O can be used for transactional memory
ReVive I/O:
Efficient Handling of I/O in Highly-Available Rollback-Recovery
ServersJun Nakano✢, Pablo Montesinos, Kourosh Gharachorloo❆, Josep Torrellas
University of Illinois at Urbana-Champaign✢IBM❆Google
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Distributed N+1 Parity in ReVive
28
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Distributed N+1 Parity in ReVive
28
Parity
Node O
Data
Node 1
Data
Node N
...
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Distributed N+1 Parity in ReVive
28
Parity
Node O
Data
Node 1
Data
Node N
...Parity Group
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Distributed N+1 Parity in ReVive
• Allocation granularity: page
• Update granularity: cache line
• Distributed parity: can recover from loss of one node
28
Parity
Node O
Data
Node 1
Data
Node N
...Parity Group
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Space Overhead
• Network PDD:
• Need to increase OS’s TCP sliding window size
• e.g., Gigabit Ethernet, T = 100 ms → 12 MB
• Disk PDD (buffering):
• Disk PDD needs private memory area for buffering
• e.g., 40 MB/s disk, T = 100 ms → 4 MB
29
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Latency-oriented workload
30
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Latency-oriented workload
30
100 200 300 400 500
800
700
600
500
400
300
200
100
0
Number of Clients
Res
ponse
Tim
e (m
s)
WebStone + Apache
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Latency-oriented workload
30
100 200 300 400 500
BaselineBuffering - 20ms CKPTBuffering - 40ms CKPTBuffering - 80ms CKPTBuffering - 160ms CKPT
800
700
600
500
400
300
200
100
0
Number of Clients
Res
ponse
Tim
e (m
s)
WebStone + Apache
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Latency-oriented workload
• Tolerable checkpoint interval is application dependent
• Response times up to 100ms are acceptable for interactive applications
30
100 200 300 400 500
BaselineBuffering - 20ms CKPTBuffering - 40ms CKPTBuffering - 80ms CKPTBuffering - 160ms CKPT
800
700
600
500
400
300
200
100
0
Number of Clients
Res
ponse
Tim
e (m
s)
WebStone + Apache
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Disk PDD Schemes
31
StallOn I/O write issue Stalls the write
After next CKPT Writes data to disk
LoggingOn I/O write issue
Copies old data elsewhere in disk Writes data in place
After next CKPT Deletes pointer to old data
BufferingOn I/O write issue Writes data to disk buffer and mem
After next CKPT Copies data from mem to disk
RenamingOn I/O write issue
Writes data to renamed disk location Saves old logical → physical mapping
After next CKPT Delete old mapping
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
• Resend packets after rollback
• TCP eliminates duplicates
32
• Can avoid saving inputs replay
• Sender timeouts and resendKernel, APP
Network PDD
CKPT CKPT
Client
Time
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
• Resend packets after rollback
• TCP eliminates duplicates
32
Kernel, APP
Network PDD
CKPT CKPT
Client
Time
• Can avoid saving inputs replay
• Sender timeouts and resendKernel, APP
Network PDD
CKPT CKPT
Client
Time
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
• Resend packets after rollback
• TCP eliminates duplicates
32
Kernel, APP
Network PDD
CKPT CKPT
Client
Time
• Can avoid saving inputs replay
• Sender timeouts and resendKernel, APP
Network PDD
CKPT CKPT
Client
Time
PKTi
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
• Resend packets after rollback
• TCP eliminates duplicates
32
Kernel, APP
Network PDD
CKPT CKPT
Client
Time
• Can avoid saving inputs replay
• Sender timeouts and resendKernel, APP
Network PDD
CKPT CKPT
Client
Time
PKTi
PKTi
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
• Resend packets after rollback
• TCP eliminates duplicates
32
Kernel, APP
Network PDD
CKPT CKPT
Client
Time
• Can avoid saving inputs replay
• Sender timeouts and resendKernel, APP
Network PDD
CKPT CKPT
Client
Time
PKTiX
PKTi
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
• Resend packets after rollback
• TCP eliminates duplicates
32
Kernel, APP
Network PDD
CKPT CKPT
Client
Time
• Can avoid saving inputs replay
• Sender timeouts and resendKernel, APP
Network PDD
CKPT CKPT
Client
Time
PKTiX
PKTi
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
• Resend packets after rollback
• TCP eliminates duplicates
32
Kernel, APP
Network PDD
CKPT CKPT
Client
Time
• Can avoid saving inputs replay
• Sender timeouts and resendKernel, APP
Network PDD
CKPT CKPT
Client
Time
PKTiX X
PKTi
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Network PDD and TCP
• Resend packets after rollback
• TCP eliminates duplicates
32
Kernel, APP
Network PDD
CKPT CKPT
Client
Time
• Can avoid saving inputs replay
• Sender timeouts and resendKernel, APP
Network PDD
CKPT CKPT
Client
Time
PKTiX X
PKTi
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
Integration with ReVive
• Simple model of ReVive overhead:
• c and r are constants (regardless of checkpoint frequency)
• For instance, c = 1 ms, r=5% and T = 100 ms give 6% overhead
33
Checkpointc [ms]
Time
overhead r
checkpoint interval T
Motivation Concept Implementation Evaluation Conclusions
HPCA 12 - ReVive I/O. Pablo Montesinos. University of Illinois.
0 20 40 60 80 100 120 140 160 180
Integration with ReVive (WebStone)
• Increase in a latency-bound workload is 2 x T (Checkpoint Interval)
• Most of the overhead comes from ReVive I/O
34
ReVive + ReVive I/O
ReVive Only
Checkpoint Interval (ms)
Incr
ease
in r
esponse
tim
e (m
s) 400
350
300
250
200
150
100
50
0
