Speculative Execution in a Distributed File System

Ed NightingalePeter ChenJason Flinn

University of Michigan

2

Motivation• Why are distributed file systems slow(er)?

– Sync n/w messages provide consistency– Sync disk writes provide safety

• Sacrifice guarantees for speed

• Can DFS can be safe, consistent and fast?– Yes! With OS support for speculative execution

3

Big Idea: Slow Way

RPC Req

Client

RPC Resp

• Guarantees without blocking I/O!

Server

Block!2) Speculate!

1) Checkpoint

Big Idea: Speculator

3) Correct?

Yes: discard ckpt.No: restore process & re-execute RPC Req

RPC Resp

RPC Req

RPC Resp

4

Conditions for Success

• Operations are highly predictable– Conflicts are rare

• Checkpoints are cheaper than network I/O– 52 µs for small process

• Computers have resources to spare– Need memory and CPU cycles for speculation

5

Outline• Motivation

• Implementing speculation

• Multi-process speculation

• Using Speculator

• Evaluation

6Undo log

Implementing SpeculationPro

cess

Checkpoint Spec

1) System call 2) Create speculation

Time

7

Speculation Success

Undo log

Checkpoint

1) System call 2) Create speculation

Proce

ss

3) Commit speculation

Time

Spec

8

Speculation Failure

Undo log

Checkpoint

1) System call 2) Create speculation

Proce

ss

3) Fail speculation

Proce

ss

Time

Spec

9

Ensuring Correctness

• Spec processes often affect external state

• Three ways to ensure correct execution– Block– Buffer– Propagate speculations (dependencies)

10

Systems Calls• Block calls that externalize state

– Allow read-only calls (e.g. getpid)– Allow calls that modify only task state (e.g. dup2)

• File system calls -- need to dig deeper– Mark file systems that support Speculator

getpidrebootmkdir

Call sys_getpid()Block until specs resolvedAllow only if fs supports Speculator

11

Output Commits

“stat worked”

“mkdir worked”

Undo log

Checkpoint

Checkpoint

Spec(stat)

Spec(mkdir)

1) sys_stat 2) sys_mkdir

Proce

ss

Time

3) Commit speculation

12

Multi-Process Speculation

• Processes often cooperate– Example: “make” forks children to compile, link, etc.– Would block if speculation limited to one task

• Allow kernel objects to have speculative state– Examples: inodes, signals, pipes, Unix sockets, etc.– Propagate dependencies among objects– Objects rolled back to prior states when specs fail

13

Spec 1Spec 1

Multi-Process Speculation

Spec 2

pid 8001

Checkpoint

Checkpoint

inode 3456

Chown-1

Write-1

pid 8000

CheckpointCheckpoint

Checkpoint

Chown-1

Write-1

14

Multi-Process Speculation

• What we handle:– DFS objects, RAMFS, Ext3, Pipes & FIFOs– Unix Sockets, Signals, Fork & Exit

• What we don’t (i.e. we block)– System V IPC– Multi-process write-shared memory

15

Outline• Motivation

• Implementing speculation

• Multi-process speculation

• Using Speculator

• Evaluation

16

Example: NFSv3 LinuxClient 1 Client 2Server

Open BGetattr

Modify BWrite

Commit

17

Example: SpecNFS

Modify Bspeculate

GetattrOpen B

speculate

Open BGetattrspeculate

Write+Commit

Client 1 Client 2Server

18

Problem: Mutating Operations

• bar depends on cat foo• What does client 2 view in bar?

Client 1

1. cat foo > bar

Client 2

2. cat bar

19

Solution: Mutating Operations• Server determines speculation success/failure

– State at server never speculative

• Send server hypothesis speculation based on– List of speculations an operation depends on

• Requires server to track failed speculations

• Requires in-order processing of messages

20

Group Commit

• Previously sequential ops now concurrent• Sync ops usually committed to disk• Speculator makes group commit possible

write

writecommit

commit

ClientClient Server Server

21

Putting it all Together: SpecNFS

• Apply Speculator to an existing file system

• Modified NFSv3 in Linux 2.4 kernel– Same RPCs issued (but many now asynchronous)– SpecNFS has same consistency, safety as NFS– Getattr, lookup, access speculate if data in cache– Create, mkdir, commit, etc. always speculate

22

Putting it all Together: BlueFS• Design a new file system for Speculator

– Single copy semantics– Synchronous I/O

• Each file, directory, etc. has version number– Incremented on each mutating op (e.g. on write)– Checked prior to all operations.– Many ops speculate and check version async

23

Outline• Motivation

• Implementing speculation

• Multi-process speculation

• Using Speculator

• Evaluation

24

Apache Benchmark

• SpecNFS up to 14 times faster

0

50

100

150

200

250

300

No delay

Tim

e (s

econ

ds)

NFSSpecNFSBlueFSext3

0

500

1000

1500

2000

2500

3000

3500

4000

4500

30 ms delay

25

The Cost of Rollback

• All files out of date SpecNFS up to 11x faster

0

20

40

60

80

100

120

140

NFS SpecNFS ext3No delay

Tim

e (s

econ

ds)

0200400600800

100012001400160018002000

NFS SpecNFS ext330ms delay

No files invalid10% files invalid50% files invalid100% files invalid

26

Conclusion

• Speculator greatly improves performance of existing distributed file systems

• Speculator enables new file systems to be safe, consistent and fast

27

Group Commit & Sharing State

050

100150200250300350400450500

NFS SpecNFS BlueFS

0 ms delay

Tim

e (s

econ

ds)

0500

10001500200025003000350040004500

NFS SpecNFS BlueFS30ms delay

DefaultNo propNo grp commitNo grp commit & no prop

28

Apache Benchmark

0

50

100

150

200

250

300

No delay

Tim

e (s

econ

ds)

0500

10001500200025003000350040004500

30ms delay

Remove MakeConfigure Untar

29

Related Work• Chang & Gibson, Fraser & Chang

– Speculative pre-fetching

• Time Warp– Virtual Time: distributed simulations

• Hardware branch prediction

• Transactional file systems