Optimistic Crash Consistency

Vijay ChidambaramThanumalayan Sankaranarayana Pillai

Andrea Arpaci-DusseauRemzi Arpaci-Dusseau

SOSP 13

Crash Consistency Problem

Single file-system operation updates multiple on-disk data structures

System may crash in middle of updates

File-system is partially (incorrectly) updated

SOSP 13

Performance OR ConsistencyCrash-consistency solutions degrade performance

Users forced to choose between high performance and strong consistency

- Performance differs by 10x for some workloads

Many users choose performance

- ext3 default configuration did not guarantee crash consistency for many years

- Mac OSX fsync() does not ensure data is safe

“The Fast drives out the Slow even if the Fast is wrong” - Kahan

SOSP 13

Ordering and Durability

Crash consistency is built upon ordered writes

File systems conflate ordering and durability

- Ideal: {A, B} -> {C} (made durable later)

- Current scenario

• {A, B} durable

• {C} durable

Inefficient when only ordering is required

SOSP 13

Can a file system provide both

high performance and strong consistency?

Is there a middle ground between:high performance but no consistency

strong consistency but low performance?

SOSP 13

Our solutionOptimistic File System (OptFS)

Journaling file system that provides performance and consistency

by decoupling ordering and durability

Such decoupling allows OptFS to trade freshness for performance while

maintaining crash consistency

SOSP 13

Results

Techniques: checksums, delayed writes, etc.

OptFS provides strong consistency

- Equivalent to ext4 data journaling

OptFS improves performance significantly

- 10x better than ext4 on some workloads

New primitive osync() provides ordering among writes at high performance

SOSP 13

Outline

Introduction

Ordering and Durability in Journaling

Optimistic File System

Results

Conclusion

SOSP 13

OutlineIntroduction

Ordering and Durability in Journaling- Journaling Overview

- Realizing Ordering on Disks

- Journaling without Ordering

Results

Conclusion9

SOSP 13

Journaling Overview

Before updating file system, write note describing update

Make sure note is safely on disk

Once note is safe, update file system

- If interrupted, read note and redo updates

SOSP 13 Journal

Workload: Creating and writing to a fileJournaling protocol (ordered journaling)

Journaling Overview

FILE SYSTEM

APPLICATIONMETADATADATA

SOSP 13 Journal

Workload: Creating and writing to a fileJournaling protocol (ordered journaling)- Data write (D)

Journaling Overview

FILE SYSTEM

APPLICATION

METADATADATA

SOSP 13 Journal

Journaling Overview

FILE SYSTEM

APPLICATION

METADATADATA

SOSP 13 Journal

- Logging Metadata (JM)

Journaling Overview

FILE SYSTEM

APPLICATION

METADATADATA

SOSP 13 Journal

Journaling Overview

FILE SYSTEM

APPLICATION

METADATADATA

SOSP 13 Journal

- Logging Commit (JC)

Journaling Overview

FILE SYSTEM

APPLICATION

METADATADATA

SOSP 13 Journal

Journaling Overview

FILE SYSTEM

APPLICATION

D JM JC

METADATADATA

SOSP 13 Journal

- Checkpointing (M)

Journaling Overview

FILE SYSTEM

APPLICATION

D JM JC

METADATADATA

SOSP 13 Journal

- Checkpointing (M)

Journaling Overview

FILE SYSTEM

APPLICATION

D JM JCM

METADATADATA

SOSP 13

OutlineIntroduction

Results

Conclusion12

SOSP 13

How Writes are Ordered

Disk Cache

Disk Platter

Original Disks

Disks with Write Buffers

SOSP 13 Journal

Journaling with Flushes

FILE SYSTEM

DISK CACHE

APPLICATIONMETADATADATA

DISK PLATTER

Journaling protocol- Data write (D)

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

METADATADATA

DISK PLATTER

FLUSH FLUSH

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC

METADATADATA

DISK PLATTER

FLUSH FLUSH

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC

METADATADATA

DISK PLATTER

FLUSH FLUSH

- Checkpointing (M)

SOSP 13

OutlineIntroduction

Results

Conclusion15

SOSP 13

Journaling without Ordering

Practitioners turn off flushes due to performance degradation- Ex: ext3 by default did not enable flushes for

many years

Observe crashes do not cause inconsistency for some workloads

We term this probabilistic crash consistency- Studied in detail

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC M

METADATADATA

DISK PLATTER

FLUSH FLUSH

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC M

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC M

METADATADATA

DISK PLATTER

Without flushes, blocks may be reordered

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

METADATADATA

DISK PLATTER

Without flushes, blocks may be reordered- Ex: JC and JM written first as disk head near journal

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JCM

METADATADATA

DISK PLATTER

Without flushes, blocks may be reordered- Ex: JC and JM written first as disk head near journal

SOSP 13

Probabilistic Crash Consistency

D JM JC M TimeMEMORY

SOSP 13

DISK JC

SOSP 13

DISK D JMJC M

SOSP 13

Re-ordering leads to windows of vulnerability

D JM JC M Time

Window

Total I/O Time

P-inconsistency = Time in window(s) / Total I/O Time

MEMORY

DISK D JMJC M

SOSP 13

Probabilistic Crash Consistencyp-inconsistency for different workloads- Read-heavy workloads have low p-inconsistency

- Database workloads have high p-inconsistency

See paper for detailed study- Factors that affect p-inconsistency

Turning off flushing provides performance, but does not ensure consistency

Additional techniques required to obtain both performance and consistency

SOSP 13

OutlineIntroduction

Optimistic File System- Overview

- Handling Re-Ordering

- New File-system Primitives

Results

Conclusion20

SOSP 13

Optimistic File SystemAchieves both performance and consistency by trading on new axis

Freshness indicates how up-to-date state is after a crash

OptFS provides strong consistency while trading freshness for increased performance

State 1 State 2 State 3 State 4Xext4OptFS

SOSP 13

Eliminates flushes in the common case

Blocks may be re-ordered without flushes

Optimistic Crash Consistency handles re-orderings with different techniques- Some re-orderings are detected after crash

- Some re-orderings are prevented from occurring

SOSP 13

Modified Disk InterfaceAsynchronous Durability Notifications (ADN) signal when block is made durable

Disk Cache

Disk Platter

SOSP 13

Modified Disk Interface

ADNs increase disk freedom- Blocks can be destaged in any order

- Blocks can be destaged at any time

- Only requirement is to inform upper layer

OptFS uses ADNs to control what blocks are dirty at the same time in disk cache

- Re-ordering can only happen among these blocks

SOSP 13

OutlineIntroduction

Results

Conclusion25

SOSP 13 Journal

Handling Re-Ordering: Removing Flush #1

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC M

METADATADATA

DISK PLATTER

Flush after JM is removed- Checksums used to handle reordering

SOSP 13

Technique #1: Checksums

JC could be re-ordered before D or JM

D JM JC M

Re-ordering detected using checksums- Computed over data and metadata

- Checked during recovery

- Mismatch indicates blocks were lost during crash

SOSP 13

Handling Re-Ordering: Removing Flush #2

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC M

METADATADATA

DISK PLATTER

Flush after JC is removed- Delayed writes used to prevent reordering

Journal

SOSP 13

Technique #2: Delayed Writes

M could be re-ordered before D or JM or JC

D JM JC M

Re-ordering prevented using delayed writes- Wait until ADN arrive for D, JM, and JC

- Then issue M to disk cache

- Invariant: D/JM/JC and M never dirty in cache together

D JM JC

SOSP 13 Journal

Optimistic Journaling

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC M

METADATADATA

DISK PLATTER

Checksums and Delayed Writes handle reordering from removing flushes

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC M

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC M

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC

METADATADATA

DISK PLATTER

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC

METADATADATA

DISK PLATTER

D JM JC

SOSP 13 Journal

FILE SYSTEM

DISK CACHE

APPLICATION

D JM JC

METADATADATA

DISK PLATTER

D JM JC

SOSP 13

Optimistic Techniques

Other Techniques - In-order journal recovery and release

- Reuse after notification

- Selective data journaling

See paper for more details

SOSP 13

OutlineIntroduction

Results

Conclusion32

SOSP 13

File-system Primitives

write(log)

write(header)

fsync(log)

fsync(header)

write(log)

write(header)

osync(log)

dsync(header)

fsync() provides ordering and durability

OptFS splits fsync()- osync() for only ordering and high performance

- dsync() for durability

Primitives can increase performance- Ex: SQLite

SOSP 13

ImplementationOptFS based on ext4 code

- Around 3000 lines of modified/added code

Required modifications to- Journaling layer

- Virtual Memory subsystem

ADNs were emulated using timeouts- Block received by disk at time T

- Block durable at time T+D

- D = 30 s in our implementation (conservative)34

SOSP 13

Outline

Introduction

Results

Conclusion

SOSP 13

Evaluation

Does OptFS preserve file-system consistency after crashes?- OptFS consistent after 400 random crashes

How does OptFS perform?- OptFS 4-10x better than ext4 with flushes

Can meaningful application-level consistency be built on top of OptFS?- Studied gedit and SQLite on OptFS

SOSP 13

Testing Application-Level Consistency

Methodology- Start from initial disk image

- Run application

• Replace fsync() with osync()

• Trace writes

- Re-order writes

- Drop writes after random point

- Replay writes on initial disk image

- Examine application state on new image

SOSP 13

SQLite Consistency

Initial Image Final ImageW1 W2 W3 W4

SOSP 13

SQLite Consistency

Initial ImageW1 W2 W3 W4

SOSP 13

SQLite Consistency

Initial Image

W1 W2W3 W4

SOSP 13

SQLite Consistency

Initial Image

W1 W2W3 W4

SOSP 13

SQLite Consistency

Initial Image

W1 W2W3

Crashed Image

ext4 without flushes 73%

SOSP 13

SQLite Consistency

Initial Image

W1 W2W3

Zero inconsistencies with OptFS

or ext4 with flushes

SOSP 13

SQLite Consistency

Initial Image

W1 W2W3

Final Image

ext4 with flushes 50%50%

SOSP 13

SQLite Consistency

Initial Image

W1 W2W3

Final Image

ext4 with flushes 50%50%OptFS 24%76%

osync() changes semantics from ACID to ACI-(Eventual Durability)

SOSP 13

SQLite Consistency

Initial Image

W1 W2W3

Final Image

Time150 ms15 ms

SOSP 13

SQLite Consistency

Initial Image

W1 W2W3

Final Image

Time150 ms15 ms

SQLite is able to provide ACI semantics with osync(), at 10x performance

SOSP 13

Outline

Introduction

Results

Conclusion

SOSP 13

SummaryProblem: providing both performance and consistency

Solution: decoupling ordering and durability in OptFS

Eventual Durability maintains consistency while trading freshness for increased performance

osync() provides a cheap primitive to order application writes

SOSP 13

Conclusion

Storage-stack layers are increasing- 18 layers between application and storage [Thereska13]

- Interfaces that provide freedom to each layer are the way forward

First impulse: trade consistency for performance- Trade-off not required in distributed systems [Escriva12]

- By trading freshness, we can obtain both consistency and high performance

SOSP 13

Thank You

Source code http://research.cs.wisc.edu/adsl/Software/optfs/

http://github.com/vijay03/optfs

Questions?

Optimistic Crash Consistency

Documents

Transcript of Optimistic Crash Consistency

Webinar: Eventual Consistency != Hopeful Consistency

Consistency in Optimistic Replication Systems - … in Optimistic Replication Systems by ... One advantage of epidemic propagation ... The timestamp can come from a physical clock,

Strata: A Cross Media File Systemawang/courses/cop5611_s2020/strata.pdfIntuitive crash consistency. 13. Strata: LibFS. Kernel-bypass NVM. Synchronous IO • When each system call returns:

Software-Transparent Crash Consistency for Persistent Memory · The Persistent Memory Manager (PMM) ! Exposes a load/store interface to access persistent data " Applications can directly

(Too) Optimistic About Optimism: The Belief That Optimism Improves Performance Files/(Too) Optimistic about... · (Too) Optimistic About Optimism: The Belief That Optimism Improves

Optimistic Work Cultures

secrets of optimistic people

Consistency & Replication Consistency & replicationmcrane/CA464/Slides/slidesLecture10.pdf · Consistency & Replication Data-Centric Consistency Models Data-centric consistency models

Crash Consistency: FSCK and Journaling - UW-Madison Computer

Optimistic Fault Tolerance - SIGTBD

Optimistic Virtual Synchrony

The Effects of Conference Call Content on Market ......suggesting overly optimistic tone increases investors’ uncertainty about the firm’s crash risk. This result is more pronounced

Finding Crash-Consistency Bugs with Bounded Black-Box Testing · Bounded Black-Box Crash Testing (B3) Focus on reproducible bugs resulting in metadata corruption, data loss. Found

Witcher: Systematic Crash Consistency Testing for Non ...

Application Crash Consistency and Performance with CCFSvijay/papers/fast17-c2fs.pdf · by experienced developers, such as Google’s LevelDB and Linus Torvalds’s Git. However, in

Optimistic in nature

Replication: optimistic approaches

Crash Consistency: FSCK and Journalingnyx.skku.ac.kr/wp-content/uploads/2018/03/OS17-Crash... · 2018-05-17 · Dongkun Shin, SKKU 5 Crash Scenarios - Two writes succeed • I[v2]

London Pessimistic Optimistic

21- Implementing Optimistic Concurrency