CS 162Section 10

Two-phase commitFault-tolerant computing

Sec 10.24/14/2014 CS162 ©UCB Spring 2014

Administrivia

• Project 3 code due Thursday 4/17 by 11:59PM

• Project 4 design due date– Thursday 4/24 by 11:59PM

• Midterm II is April 28th 4-5:30pm in 245 Li Ka Shing and 100 GPB– Covers Lectures #13-24– Closed book and notes, no calculators– One double-sides handwritten page of notes allowed– Review session: Fri Apr 25 4-6pm in 245 Li Ka Shing

Sec 10.34/14/2014 CS162 ©UCB Spring 2014

QUIZ

Sec 10.44/14/2014 CS162 ©UCB Spring 2014

QuizTrue/False1. Two-phase commit is used to guarantee consistency. False2. You always need 2PC if you want multiple systems to stay

consistent. False (ex: use consistent client-side hashing to put keys amongst slaves, with no replication.)

3. If a master server comes awake after crashing in the WAIT state, it resends VOTE_REQ in order to recount the slaves. False

4. Increasing mean-time-to-repair decreases availability (as defined in lecture). True

5. Bohrbugs won’t be fixed by restarting a task or system. True

Short answer5. What was the purpose of the TLS heartbeat, target of the

Heartbleed attack? Make sure NATs/firewalls in the middle don’t shut down the connection.

Sec 10.54/14/2014 CS162 ©UCB Spring 2014

TWO-PHASE COMMIT

Sec 10.64/14/2014 CS162 ©UCB Spring 2014

Durability and Atomicity, distributed• How do you make sure transaction results persist in

the face of failures (e.g., disk failures)?

• Replicate database– Commit transaction to each replica

• What happens if you have failures during a transaction commit?– Need to ensure atomicity: either transaction is committed

on all replicas or none at all

• How can we replicate with atomicity?

Sec 10.74/14/2014 CS162 ©UCB Spring 2014

Two-Phase Commit

Coordinator sends VOTE-REQ to all workers

– Wait for VOTE-REQ from coordinator– If ready, send VOTE-COMMIT to

coordinator– If not ready, send VOTE-ABORT to

coordinator– And immediately abort

– If receive VOTE-COMMIT from all N workers, send GLOBAL-COMMIT to all workers

– If doesn’t receive VOTE-COMMIT from all N workers, send GLOBAL-ABORT to all workers

– If receive GLOBAL-COMMIT then commit

– If receive GLOBAL-ABORT then abort

Coordinator Algorithm Worker Algorithm

Sec 10.84/14/2014 CS162 ©UCB Spring 2014

Worker, Master statesINIT

WAIT

ABORT COMMIT

Recv: STARTSend: VOTE-REQ

Recv: VOTE-ABORTSend: GLOBAL-ABORT

Recv: VOTE-COMMITSend: GLOBAL-COMMIT

INIT

READY

ABORT COMMIT

Recv: VOTE-REQSend: VOTE-ABORT

Recv: VOTE-REQSend: VOTE-COMMIT

Recv: GLOBAL-ABORT Recv: GLOBAL-COMMIT

Master

Worker

Sec 10.94/14/2014 CS162 ©UCB Spring 2014

Failure Free Example Execution

coordinator

worker 1

time

VOTE-REQ

VOTE-COMMIT

GLOBAL-COMMIT

worker 2

worker 3

Sec 10.104/14/2014 CS162 ©UCB Spring 2014

Dealing with Worker Failures

• How to deal with worker failures?– Failure only affects states in which the node is waiting for

messages– Coordinator only waits for votes in “WAIT” state– In WAIT, if doesn’t receive

N votes, it times out and sendsGLOBAL-ABORT

INIT

WAIT

ABORT COMMIT

Recv: STARTSend: VOTE-REQ

Recv: VOTE-ABORTSend: GLOBAL-ABORT

Recv: VOTE-COMMITSend: GLOBAL-COMMIT

Sec 10.114/14/2014 CS162 ©UCB Spring 2014

Dealing with Coordinator Failure

• How to deal with coordinator failures?– worker waits for VOTE-REQ in INIT

» Worker can time out and abort (coordinator handles it)– worker waits for GLOBAL-* message in READY

» If coordinator fails, workers mustBLOCK waiting for coordinatorto recover and sendGLOBAL_* message

INIT

READY

ABORT COMMIT

Recv: VOTE-REQSend: VOTE-ABORT

Recv: VOTE-REQSend: VOTE-COMMIT

Recv: GLOBAL-ABORT Recv: GLOBAL-COMMIT

Sec 10.124/14/2014 CS162 ©UCB Spring 2014

Example of Coordinator Failure

VOTE-REQ

VOTE-COMMIT

INIT

READY

ABORT COMM

block waiting for coordinator

restarted

GLOBAL-ABORT

coordinator

worker 1

worker 2

worker 3

Sec 10.134/14/2014 CS162 ©UCB Spring 2014

Remembering Where We Were (Durability)• All nodes use stable storage* to store which state they are

in

• Upon recovery, it can restore state and resume:– Coordinator aborts in INIT, WAIT, or ABORT– Coordinator commits in COMMIT– Worker aborts in INIT, ABORT– Worker commits in COMMIT– Worker asks Coordinator in READY

* - stable storage is non-volatile storage (e.g. backed by disk) that guarantees atomic writes.

Sec 10.144/14/2014 CS162 ©UCB Spring 2014

FAULT-TOLERANT COMPUTING

Lec 20.154/14/2014 CS162 ©UCB Spring 2014

Dependability: The 3 ITIES

• Reliability / Integrity: does the right thing. (Need large MTBF)

• Availability: does it now. (Need small MTTR

MTBF+MTTR)

• System Availability:if 90% of terminals up & 99% of DB up?

(=> 89% of transactions are serviced on time)

SecurityIntegrityReliability

Availability

MTBF or MTTF = Mean Time Between (To) FailureMTTR = Mean Time To Repair (see next slide)

Sec 10.164/14/2014 CS162 ©UCB Spring 2014

Mean Time to Recovery• Critical time as further failures can occur during recovery

• Total Outage duration (MTTR) =Time to Detect (need good monitoring)+ Time to Diagnose (need good docs/ops, best practices)+ Time to Decide (need good org/leader, best practices)+ Time to Act (need good execution!)

Sec 10.174/14/2014 CS162 ©UCB Spring 2014

Traditional Fault Tolerance Techniques• Fail fast modules: work or stop

• Spare modules: yield instant repair time

• Process/Server pairs: Mask HW and SW faults

• Transactions: yields ACID semantics (simple fault model)

Sec 10.184/14/2014 CS162 ©UCB Spring 2014

Fail-Fast is Good, but Repair is Needed

Improving either MTTR or MTBF gives benefitSimple redundancy does not help much (can actually hurt!)

Lifecycle of a modulefail-fast gives short fault latency

High Availability is low UN-Availability

Unavailability ~ MTTR MTBF

X XFault Detect

RepairReturn

Sec 10.194/14/2014 CS162 ©UCB Spring 2014

Fail-Fast and High-Availability ExecutionProcess Pairs: Instant repair

Use Defensive programming to make a process fail-fastHave separate backup process ready to “take over” if primary faults

• SW fault is a Bohrbug no repair“wait for the next release” or “get an emergency bug fix” or“get a new vendor”

• SW fault is a Heisenbug restart process“reboot and retry”

• Yields millisecond repair times

• Tolerates some HW faults

SESSIONPRIMARYPROCESS

BACKUPPROCESS

STATEINFORMATION

LOGICAL PROCESS = PROCESS PAIR