Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& •...
Transcript of Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& •...
![Page 1: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/1.jpg)
Ge#ng Started with NVM Programming
• NVM emula7on • File systems – BPFS – PMFS
• Persistent heaps – Mnemosyne – libpmem
![Page 2: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/2.jpg)
Emula7on Techniques
• Performance (based on DRAM) – Latency – Bandwidth
• Func7onality – Persistent memory (NVM plugged on memory bus) – Power failures
![Page 3: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/3.jpg)
Intel Hardware Emulator
• Latency emula7on – Uses special microcode to inject addi7onal stall cycles every few memory accesses
• Bandwidth emula7on – Programs memory controller to throNle bandwidth
• Available through Intel
CPU $ MC
DRAM
NVM (DRAM)
addi7onal stalls
![Page 4: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/4.jpg)
SoQware-‐created Delays
• Inject delays via: – clflush + delay for slower memory writes – RAM-‐disk + delay for slower block reads/writes
• Create delays using a spin-‐loop
• Available through Mnemosyne library
for (start = RDTSCP; // read cpu *mestamp counter RDTSCP – start < delay; )
![Page 5: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/5.jpg)
PCMSIM: NVM-‐disk Emula7on
• Emulates a PCM-‐disk by slowing down DRAM – Implemented as a Linux device-‐driver – Exposes PCM-‐disk as /dev/pcm0
• Supports unmodified exis7ng disk-‐based file systems (e.g. ext4)
• Available at: hNps://code.google.com/p/pcmsim/
![Page 6: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/6.jpg)
Func7onality
• Emula7ng PM: NVM plugged on memory bus – Use a memory-‐mapped file – Reserve DRAM when the kernel boots
• Valida7ng soQware against power-‐failures – Kill the program using SIGKILL (but cache survives) – Unplug flash-‐backed NV-‐DIMMs – Inject failures via binary instrumenta7on (e.g., Yat)
![Page 7: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/7.jpg)
Ecosystem Overview
Applica7on
Intel STM Compiler libpmem
PMFS BPFS
Emulated Persistent Memory (via DRAM)
read/ write
read/ write
mmap mmap mmap
Mnemosyne
Direct to PM (region manager)
Direct to PM (XIP/DAX)
user
kernel
read/ write
ext4
PCMSIM (NVM-‐disk)
load/store Block I/O
![Page 8: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/8.jpg)
BPFS (FUSE-‐based version)
• Supports crash-‐consistency via copy-‐on write (CoW) – Short circuit CoW – No epoch ordering
• Emulates PM through a memory-‐mapped file
APP
VFS
BPFS
libfuse
mmap-‐ed file
read/ write
NVM
• Available at: hNp://bpfs.cs.ucla.edu
![Page 9: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/9.jpg)
Ge#ng started with BPFS
• Download > git clone git://bpfs.cs.ucla.edu/bpfs/bpfs
• Build (requires libfuse-‐dev installed) > make
• Create emulated-‐NVM backing store file > dd if=/dev/zero of=bpram.img bs=1M count=$N
• Format > ./mkfs.bpfs bpram.img $MNT
• Mount > ./bpfs –f bpram.mnt $MNT
![Page 10: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/10.jpg)
PMFS
• Supports crash-‐consistency via metadata journaling – Ordering: clflushopt/pcommit
• Enables direct access to PM via XIP/DAX
• Emulates PM by reserving physically con7guous DRAM
• Available at: hNps://github.com/linux-‐pmfs/pmfs
Applica7on
NVM DRAM
read/ write
mmap
Direct (XIP/DAX)
VFS
PMFS
![Page 11: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/11.jpg)
Ge#ng Started with PMFS
• Download Linux kernel 3.11 including PMFS > git clone https://github.com/linux-‐pmfs/pmfs.git
• Build > make menuconfig # Enable PMFS and PMFS XIP > make && make modules_install install
• Reserve DRAM for use as NVM via boot op7on – e.g. memmap=2G$4G: reserves 2G star7ng at 4G
• Mount > mount –t pmfs –o physaddrs=0x100000000,init=2G none /mnt/pmfs
![Page 12: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/12.jpg)
Persistent Heaps
Applica7on
Intel STM Compiler libpmem
PMFS BPFS
Emulated Persistent Memory (via DRAM)
read/ write
read/ write
mmap mmap
Mnemosyne
Direct to PM (region manager)
Direct to PM (XIP/DAX)
user
kernel
read/ write
ext4
PCMSIM (NVM-‐disk)
load/store
![Page 13: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/13.jpg)
Mnemosyne
• Mnemosyne API supports – Persistent memory regions – Ordering primi7ves – Log and durable memory transac7ons
• Available at: hNp://research.cs.wisc.edu/sonar/projects/mnemosyne
![Page 14: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/14.jpg)
Mnemosyne Components Class API Module Persistent regions psta7c var
pmap(addr, len, prot, flags) punmap(addr, len) type persistent * ptr
libmcore
Persistent heap pmalloc(sz, ptr) pfree(ptr)
libpmalloc
Ordering primi7ves flush(addr) store(addr, val) wtstore(addr, val) fence()
x86 instruc>ons (mnemosyne.h)
Log log_create(flags, cbf) log_append(rec) log_flush() log_truncate()
libmcore
Durable Transac7ons patomic { … }
libmtm + Intel STM Compiler
Alloca7o
n Co
nsisten
t Upd
ates
![Page 15: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/15.jpg)
Ge#ng Started with Mnemosyne
• Download Mnemosyne and prerequisites – hNp://research.cs.wisc.edu/sonar/projects/mnemosyne
• Build > cd $MNEMOSYNE/usermode > scons
• Try > scons -‐-‐build-‐bench=kvstore
• Full documenta7on available online
![Page 16: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/16.jpg)
libpmem
• Supports SNIA NVM API (Mnemosyne like) • Builds on Linux Direct Access (DAX) • Provides a collec7on of libraries – libpmem: low-‐level persistent memory support – libpmemlog: pmem-‐resident log file – libpmemblk: arrays of pmem-‐resident blocks with atomic updates
– libpmemobj: transac7onal object store (under dev)
• Available at: hNp://pmem.io
![Page 17: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/17.jpg)
Ge#ng Started with libpmem
• Install prerequisites – Install PMFS – Install libuuid: sudo apt-‐get install uuid-‐dev
• Download NVM library > git clone https://github.com/pmem/nvml.git
• Build library > make
![Page 18: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/18.jpg)
Example: Circular Queue if ((fd = open(“myqueue", O_RDWR)) < 0) { /* allocate queue */ fd = open(“myqueue", O_CREAT|O_RDWR, 0666) posix_fallocate(fd, 0, QUEUE_MAX_LEN * QUEUE_ITEM_SIZE) fsync(fd); } /* memory map it */ struct queue_s* queue = pmem_map(fd); /* initialize queue (fail-‐safe against partial init) */ if (queue-‐>valid == 0) { queue-‐>head = queue-‐>tail = 0; pmem_persist(queue, sizeof(*queue)); queue-‐>valid = 1; pmem_persist(&queue-‐>valid, sizeof(queue-‐>valid)); }
Force changes to NVM
![Page 19: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/19.jpg)
Example: Circular Queue enqueue(queue_t* queue, queue_item_t* data) { if ((queue-‐>tail + 1) % QUEUE_MAX_LEN != queue-‐>head) { memcpy(queue-‐>buffer[queue-‐>tail], data, QUEUE_ITEM_SIZE); pmem_flush(&queue-‐>buffer[queue-‐>tail], QUEUE_ITEM_SIZE); pmem_drain(); queue-‐>tail = (tail + 1) % QUEUE_MAX_LEN; pmem_flush(&queue-‐>tail, sizeof(queue-‐>tail)); pmem_drain(); } }
Flush processor caches
Wait for pmem stores to drain from HW buffers
Equivalent to pmem_persist
![Page 20: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/20.jpg)
Adap7ng Disk-‐based Programs to NVM
• Workloads – TokyoCabinet: Key-‐value store – OpenLDAP: Directory service
• Approaches – Run on ext4 + NVM-‐disk – Run on PMFS/BPFS – Convert to use a persistent heap (e.g. Mnemosyne)
![Page 21: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/21.jpg)
TokyoCabinet: Key-‐Value Store
• Original: msyncs B-‐tree to a mmap’d file • Modified: keeps B-‐tree in persistent memory
Original version
rec = TCMALLOC(sizeof(*nrec));
pmalloc(&rec, sizeof(*nrec));
patomic { tcbdbput(bdb, &obj-‐>key, ...); }
tcbdbput(bdb, &obj-‐>key, ...); tcbdbsync(bdb);
pstatic TCBDB* bdb; tcbdbopen(bdb, ...);
Mnemosyne version
![Page 22: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/22.jpg)
OpenLDAP: Directory Service
• Original: stores dir-‐entries in Berkeley DB
• Modified: keeps dir-‐entries in persistent memory
Directory
Berkeley DB In-‐memory directory Request handler
In-‐NVM directory Request handler
![Page 23: Geng&Started&with&NVM Programming& · Geng&Started&with&NVM Programming& • NVMemulaon& • File&systems& – BPFS& – PMFS& • Persistentheaps& – Mnemosyne& – libpmem&](https://reader036.fdocuments.us/reader036/viewer/2022062414/5f722e2ad467525b0715e0d9/html5/thumbnails/23.jpg)
Future Plans
• Port Mnemosyne to GCC-‐TM compiler and PMFS
• Form a benchmark suite – TokyoCabinet – OpenLDAP – memcached (?) – …