Providing Fast and Safe Access to Next-Generation, Non-Volatile Memories

From ARIES to MARS:Transaction Support for Next-Generation, Solid-State DrivesJoel Coburn*, Trevor Bunker*, Meir Schwarz, Rajesh Gupta, Steven Swanson

Non-volatile Systems LaboratoryDepartment of Computer Science and EngineeringUniversity of California, San Diego* Now at Google#1

Spin-torque MRAMFaster than Flash Non-volatile Memories

Phase change memory

Memristor Flash is everywhere but has its idiosyncrasiesNew device characteristicsNearly as fast as DRAMNearly as dense as flashNon-volatileReliableApplicationsDRAM replacementsFast storage#2

Hard Drives (2006)

PCIe-Flash (2007)

PCIe-PCM (2010)PCIe-Flash (2012)

PCIe-PCM (2014?)

DDR Fast NVM (2016?)5917x 2.4x/yr7200x 2.4x/yrMore than Moores Law Performance#3Realizing the Potential of fast NVMsPhysical StorageLow-level IOFile SystemProcess IsolationApplicationsStorage ControllerNV-DIMMNV-DIMMNV-DIMMNV-DIMMNV-DIMMNV-DIMMLow-level IOFile SystemProcess IsolationApplications159832020292920LogX29WAL algorithms were designed for disk!#4Moneta-Direct SSD for Fast NVMsFPGA-based prototypeDDR2 DRAM emulates PCMPCIe: 2GB/s, full duplexOptimized kernel driver and device interfaceEliminate disk-based bottlenecks in IO stackUser-space driverEliminates OS and FS costs in the common case5s latency,1.8M IOPS for 512B requests

[SC 2010, Micro 2010, ASPLOS 2012]#5Characteristics of Fast SSDsDiskMonetaLatency (4KB)7000s7sBandwidth (4KB)2.6MB/s1700MB/sSequential/random performance~100:11:1Minimum request size/alignmentBlockByteParallelism 164Internal/external bandwidth1:18:1#6Existing Support for TransactionsDisk-based systemsWrite-ahead logging approaches: ARIES [TODS 92], Stasis [OSDI 06], Segment-based recovery [VLDB 09], Aether [VLDB 10]Device/HW support: Logical Disk [SOSP 93], Atomic Recovery Units [ICDCS 96], Mime [HPL-TR 92]Shadow paging in file systems: ZFS, WAFL

Non-volatile main memoryPersistent regions: RVM [TOCS 94], Rio Vista [SOSP 97]Programming support: Mnemosyne, NV-heaps [ASPLOS 11]

Flash-based SSDsTransactional Flash [OSDI 08]FusionIOs AtomicWrite [HPCA 11]#7ARIES: Write-Ahead Logging Recovery Algorithm for DatabasesFeatureBenefit(s)Flexible storage managementSupports varying length data and high concurrencyFine-grained lockingHigh concurrencyPartial rollbacks via savepointsRobust and efficient transactionsRecovery independenceSimple and robust recoveryOperation loggingHigh concurrency lock modesFast, flexible, and scalable ACID transactions#8ARIES Disk-Centric DesignDesign DecisionAdvantagesHow?No-force

Eliminate synchronous random writesFlush redo log entries to storage on commitSteal

Reclaim buffer space (scalability)Eliminate random writesAvoid false conflicts on pagesWrite undo log entries before writing back dirty pagesPagesSimplify recovery and buffer managementMatch the semantics of diskAll updates are to pagesPage writes are atomicLog Sequence Numbers (LSNs)Simplify recoveryEnable features like operation loggingLSNs provide an ordering on updatesGood for disk, not good for fast SSDs#9

MARS:Modified ARIES Redesigned for SSDsMoneta-Direct DriverStorage ManagerMoneta-Direct SSDApplicationsKernel IOFile SystemSimplified ARIES Replacement+Flexible software interface+Hardware supportEditable Atomic Writes#10Editable Atomic Writes (EAWs)Atomic { Write A Write B Write C If(x) Write A }LogDataStorageABCAWrite the logCommitApplications can access and edit the log prior to commit.Hardware copies data in-place.#11AABCEditable Atomic Write ExecutionStorageTransaction TableMetadataFileBLogFileDataFileMemoryLogWrite(t1,memA,dataA,logA);LogWrite(t1,memB,dataB,logB);LogWrite(t1,memC,dataC,logC);If(x) Write(memA,logA);Commit(t1);// WriteBack(t1);

PENDINGCCOMMITTEDFREE0

63AAAAABC#12Designing MARS for Fast NVMsNo-force

Steal

Pages

LSNsPerform write backs in hardware at the memory controllersHardware does in-place updatesEliminate undo loggingLog always holds latest copySoftware sees contiguous objectsHardware manages the layout of objects across memory controllers Hardware maintains ordering with commit sequence numbers#13MARS Features using EAWsFeatureProvided by MARS?Flexible storage managementFine-grained lockingPartial rollbacks via savepointsRecovery independenceOperation loggingN/A#148 GB8 GBLogger

EAW Hardware ArchitectureRingControlTransferBuffersDMAControlScoreboardHost viaPIOHost viaDMAReqQueueRing (4 GB/s)8 GBLogger8 GBLogger8 GBLoggerLogger8 GBLogger8 GBLogger8 GBLoggerTIDManagerTagRenamerPermCheckTID StatusReq Status2-phase commit protocolCommitPendPendPendCommCommCommWrite back AckFreeFreeFree#15Latency Breakdown

Up to 3x faster than software only#16Bandwidth Comparison2 to 3.8x improvement #17Internal Memory Bandwidth3x bandwidth#18MemcacheDB: Persistent Key Value Store1.7x faster than SoftAtomic, 3.8x faster than BDB#19Comparison of MARS and ARIES4x throughput improvement and better scalability#20Conclusions from MARSMARS: Redesign of write-ahead logging for NVMsProvides the features of ARIES but none of the disk-related overheads in a database storage managerEditable Atomic Writes (EAWs)Makes the log accessible and editable prior to commitMinimizes the cost of atomicity and durabilityOffloads logging, commit, and write back to hardwareMARS achieves 4x the performance of ARIESReduces latency and required host/device bandwidth

#21Thank you!Any questions?

#22