An Empirical Study of File Systems on NVM - MSST...

An Empirical Study of File Systems on NVMPriya Sehgal, Sourav Basu, Kiran Srinivasan, Kaladhar Voruganti

NetApp Inc.

© 2015 NetApp, Inc. All rights reserved. NetApp Confidential – Limited Use 1

Motivation

Need a thin software stack to access data from fast NVM

Persistent memory abstractions

NVM optimized file systems

What characteristics of traditional block based file systems are good for NVM?

Can traditional file systems be fine tuned using mount and format options?

Can it be optimized with minor changes?

How does the performance of traditional file systems compare with NVM-optimized one?

What file system features help improve performance on NVM?


Overview

Related Work

Experimental Methodology

Experimental Results

Recommendation and Conclusion


Overview

Related Work





Related Work


POSIX interface, re-designing the file system for

NVM

POSIX library interposers or

bypass file system

New Programming Models and

Data-Structures

Adjust operating system,

storage stack or file system configurations

Related Work



NVM


bypass file system


Data-Structures



• PMFS

• SCMFS

• BPFS

Related Work



NVM


bypass file system


Data-Structures



• PMFS

• SCMFS

• BPFS

• Moneta-D

• Bankshot

• Aerie

Related Work



NVM


bypass file system


Data-Structures



• PMFS

• SCMFS

• BPFS

• Moneta-D

• Bankshot

• Aerie

• Mnemosyne

• PMem-Lib

• NV-Heaps

• NV-Tree

• CDDS

Related Work



NVM


bypass file system


Data-Structures



• PMFS

• SCMFS

• BPFS

• Moneta-D

• Bankshot

• Aerie

• Mnemosyne

• PMem-Lib

• NV-Heaps

• NV-Tree

• CDDS

• Lee et al

• Our work.

Overview

Related Work





Experimental MethodologyWorkloads


WORKLOADS

FileServer :Moderate directory depth, moderate no. of large files, data and metadata operations




WebProxy:Flat namespace, large no. of small files, data and metadata operations

WORKLOADS



WORKLOADSOLTP: data intensive





Key-Value Store: Uses Memory-Mapped Semantics (load/stores)

WORKLOADSOLTP: data intensive



Experimental MethodologyFile System Characteristics (varied using mount and format options)


Inode Structure: Linear vs B+ Tree

Block Size: Fixed vs Variable sized extent

Layout/Update: In-place vs Log-structured vs Hybrid

Allocation Strategy: Immediate vs Delayed

Parallel Allocation (Concepts like Allocation/Block group)

Journal: Ordered vs Write-Back vs Data

Execute-in-place(XIP)

Experimental MethodologyFile System Characteristics (varied using mount and format options)


Inode Structure: Linear vs B+ Tree

Block Size: Fixed vs Variable sized extent

Layout/Update: In-place vs Log-structured vs Hybrid

Allocation Strategy: Immediate vs Delayed

Parallel Allocation (Concepts like Allocation/Block group)

Journal: Ordered vs Write-Back vs Data

Execute-in-place(XIP)

File Systems Evaluated

Ext2, Ext3, Ext4, XFS, F2FS, NILFS2, PMFS

Experimental MethodologyExperimental Setup


RamDisk(NVM

Simulated)

File System

Workload/ Benchmark

DRAM

User Space

Kernel

Block Driver

XIP

Limitations:

• NVM characteristics not simulated

• Not considered recoverabiliy

Overview

Related Work





FileServer (Throughput)


0

20

40

60

80

100

120

140

160

op

s/s

ec (

x 1

000)

1.5x2x

1.3x

XIP support, atomic updates and fine grained

logging

Higher is better100K files of size 128KB

0

20

40

60

80

100

120

140

160

op

s/s

ec (

x 1

000)



1.4x 3x

XIP support only for data in

Ext2/3

Higher is better

0

20

40

60

80

100

120

140

160

op

s/s

ec (

x 1

000)



31%

Increased AG count increased parallelism

for meta-data intensive workloads

Higher is better

0

20

40

60

80

100

120

140

160

op

s/s

ec (

x 1

000)



F2FS solves following problems of NILFS2:• Wandering tree • Garbage collection

3.1x

Higher is better

0

20

40

60

80

100

120

140

160

op

s/s

ec

(x

100

0)

100x

41x

45x

WebProxy (Throughput)At directory depth 0.7 – Flat namespace


Hash tree (indexed directory)

missing in ext2 and pmfs

500K files of size 32KB Higher is better

WebProxy (Throughput)


0

20

40

60

80

100

120

140

160

180

200

op

s/s

ec

(x

10

00

)

First Column: 0.7 Directory Depth

Second Column: 2.1 Directory Depth

Lookup is fast for increased

directory depth

PMFS scalability affected by

allocation from single linked list

Higher is better

WebProxy (Throughput)


0

20

40

60

80

100

120

140

160

180

200

op

s/s

ec

(x

10

00

)

First Column: 0.7 Directory Depth

Second Column: 2.1 Directory Depth1.6x

Delayed Allocation not good for small

files

Higher is better

OLTP Database(TPC-C on MySQL)


0

0.2

0.4

0.6

0.8

1

1.2

No

rmalized

tp

mC

XIP-FS performs best

400 warehouse, total size 38GB Higher is better

0

0.2

0.4

0.6

0.8

1

1.2

No

rmalized

tp

mC



Next Level are buffered FS:13-22%

worse than XIP



0

0.2

0.4

0.6

0.8

1

1.2

No

rmalized

tp

mC



Next Level are buffered FS:13-22%

worse than XIP

Worst Performing FS: 32-42% worse compared to XIP



0

0.2

0.4

0.6

0.8

1

1.2

No

rmalized

tp

mC



10%

Increased AG count increases indirections

for large files

Higher is better

Key-Value Stores (YCSB on MongoDB) - LatencyWORKLOAD C


0

0.2

0.4

0.6

0.8

1

1.2

1.4

No

rmalized

Late

ncy

READ

Reduced page faults blurs

performance difference

10 million records, total size 36GB

READ=100%

Lower is better

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

No

rmalized

Late

ncy

READ UPDATE

Key-Value Stores (YCSB on MongoDB) - LatencyWORKLOAD A


Msync and Fsync is costly in non-xip file

systems

Lower is better

READ=50% UPDATE=50%

Lower performing

workload affects other workloads

Key-Value Stores (YCSB on MongoDB) - LatencyWORKLOAD E


0

0.5

1

1.5

2

2.5

3

No

rmalized

Late

ncy

SCAN INSERT

Effect of page-faults shows-up in case of

loads(SCAN)

SCAN=95% INSERT=5%

Lower is better

Overview

Related Work






Recommendation for traditional and new file systems

In-place update layout

Execute In Place

Simple and parallel allocation strategy

Fixed sized data blocks


Thank you


An Empirical Study of File Systems on NVM - MSST...

Documents

Transcript of An Empirical Study of File Systems on NVM - MSST...