PASC fault tolerance
-
Upload
marco-serafini -
Category
Technology
-
view
363 -
download
2
description
Transcript of PASC fault tolerance
Taming Data Corruptions in Distributed Systems
Marco Serafini (Yahoo! Research BCN)
Infrastructure dependability
o Service availability, data durabilityo In presence of hardware faultso Current approaches tolerate crashes
Crashes
oAssumptionso A server (process) suddenly stopso Until then, only correct steps
Time
Crash
Data corruptions
oWhat if there are data corruptions?o The state of a process may be corruptedo The process may make incorrect steps before stopping
Time
Datacorruptions
Data corruptions
oWhat if there are data corruptions?o The state of a process may be corruptedo The process may make incorrect steps before stopping
Time
Datacorruptions
NOT COVERED!
Sources of data corruptions
o Commodity disks are known to be unreliableo Faulty firmware, bad sectors etc.
oRAM: ECC errors are frequento Production machines only see detected errors
Coverage not knowno Interconnects and CPUs also fail
o Faulty drivers or bit flips
A horror storyAn 8-hour system-wide outage due to a single hardware fault
What happened?
oQuoted from the Amazon service health dashboardo “A handful of messages had a single bit corrupted”o “The message was still intelligible, but the system state
information was incorrect”o “We used MD5 checksums throughout the system (but
not) for this particular internal state information”o “(The corruption) spread throughout the system causing
the symptoms described above”
Error propagation
u
v
mout
Event handling
min
min
x
y
Eventhandling
Process i Process j
Common practice
oManual placement of ad-hoc error detection checkso Application knowledgeo Time consuming
oHard to structure without fault model
oNo error isolation guarantee
Research: Byzantine faults
oByzantine modelo Faulty nodes controlled by an adversaryo Worst-case model
11
Time
Byzantinefault
Byzantine fault model
oBlack-box model of faulty processes: adversarialoHardening for error isolation [Nysiad NSDI 2008]
o Based on state machine replicationo Replication and performance costs
Servers
Client
Agreement on requests
Byzantine faults
oByzantine hardening covers attacks and bugs…o… assuming, e.g., design diversity of replicas
o Unpractical in most systems no real adoption
Attacks
Security
Bugs
V & V
Data corruptions
ASC Hardening
A new approach to error isolation
u
v
mout
Event handling
min
min
x
y
Eventhandling
Process i Process j
1. General model of process behavior2. Arbitrary State Corruption (ASC) fault model3. Guarantee error isolation through hardening
A new approach to error isolation
u
v
mout
Event handling
min
min
x
y
Eventhandling
Process i Process j
1. General model of process behavior2. Arbitrary State Corruption (ASC) fault model3. Guarantee error isolation through hardening
with M. Correia, D. Ferro and F. Junqueira2012 Usenix Annual Technical Conference
Process and fault modelsDefining Arbitrary State Corruptions
Process model
Upon receive message <REQ, r> doif v > 5 then
u = r + v + 5;
elseu = r + v;
v = u;send <WRITE, v> to
process p
min
mout
1) Event Dispatching
2) Event Handling
3) Message sending
State
ASC fault model
oAn Arbitrary State Corruption can make a process o Crasho Assign an arbitrary value to any variableo Start the execution from an arbitrary instruction
v 5
z 10
PC 20
v 12
z 7
PC 320
Fault frequency
oOne fault for every processed input message
Upon receive message <REQ, r> doif v > 5 then
u = r + v + 5;
elseu= r + v;
v = u;send <WRITE, v> to
process p
min
mout
1) Event Dispatching
2) Event Handling
3) Message sending
State
Fault diversity
oA corrupted variable is different from its replica
oOnly holds immediately after the faulto Can be invalidated if instructions modify the variable
v 5
z 10
PC 20
v 12
z 7
PC 320
5
10
5
41
original replica original replica
Error propagation
o Fault diversity does not holdoHardening preserves diversity
u
v ?
Original ReplicaFault diversity
ASC hardeningFrom ASC faults to crashes and message omissions
From ASC to crashesoTransparent: to the hardened processo Local: no process replication on multiple machinesoUntrusted: can have faults while executing hardening
HARDENING RUNTIME
u
v
mout
Event handling
min
PASC runtime
EH1 EH2 EH3
Process state
PASC checks
PASC library
User- defined
Transparent
github.com/yahoo/pasc
Replica state
Evaluation
Hardening an echo server
o Little computation, network bound, no overheado PBFT is a reference (Nysiad not available)
Hardening State Machine Replication
+ 70 %- 15 %
Zookeeper (core)
Memory overhead
Scalability
o SimpleKV: eventually consistent store, no replicationo Scales similarly with hardeningo No server “wasted” for replication
1 3 5 70
10000
20000
30000
40000
50000
60000
70000
80000
90000
100000
PASC sKVUnprot. sKV
Number of servers
Ma
x.
thro
ug
hp
ut
(ko
ps/
sec)
PASC fault coverageo Injected random bit flips in Paxos
o Code corruptions: bytecode and binary codeo State corruptions: pointers and primitive values
Code corruptions State corruptions
Unprot PASC Unprot PASC
Undet. 3 0 93 0
Det. - 1 - 330
Crash 1640 1663 2301 2066
Not manif. 1213 1193 2843 2841
Total 2856 2856 5237 5237
Wrap up
oHardware data corruptions are a real dangero Proposed new systematic approach
o BFT not realistico Ad-hoc approaches are not systematic
oHardening algorithm for error isolation o Local: does not require replicationo Efficient: PASC-Paxos has up to 70% more throughput
than PBFTo High fault coverage
Directions
o Systematic protection of Yahoo! infrastructure against data corruptions
oASC just scratched the surface – some todoso Reduce memory footprinto Support for external memory (disks/SSDs)o Hardening of legacy codeo Theoretical foundations
Thank you