Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit...
Transcript of Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit...
![Page 1: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/1.jpg)
1
Zhou, Xueqiang (CAS AE, NSG)
Hui, Chunyang (Software Engineer, NCLG)
![Page 2: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/2.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit 2
Contents
• Evaluate performance with FIO
• OCF and its configuration
![Page 3: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/3.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit 3
FIO_plugin• Can’t see device after SPDK setup
• Use SPDK fio_plugin to evaluate performance by PCI address
• Two types:
• NVMe Driver
• Bdev
• Build fio before building SPDK with fio_plugin
![Page 4: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/4.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit
USAGE – Prepare SPDK & FIO• Prepare SPDK Repo
• git clone https://github.com/spdk/spdk.git
• cd spdk && git checkout v19.07
• git submodule update –init
• scripts/pkgdep.sh
• Prepare and Compile FIO
• git clone https://github.com/axboe/fio.git
• cd fio && git checkout fio-3.3
• ./configure
• make
![Page 5: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/5.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit
USAGE – COMPILE SPDK• Compile (OCF is a submodule of SPDK)
• cd spdk
• ./configure --with-fio=/home/fio --with-ocf
• “with-fio” is the directory where you clone and compile FIO
• make
![Page 6: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/6.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit 8
Warm up: FIO with NVME• Path/to/spdk/examples/nvme/fio_plugin/
• example_config.fio: ioengine = spdk
• LD_PRELOAD=<path to spdk repo>/examples/nvme/fio_plugin/fio_plugin fio
• Directly used to evaluate NVMe disk/driver performance
![Page 7: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/7.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit 9
FIO with bdev• Path/to/spdk/examples/bdev/fio_plugin/
• example_config.fio: ioengine, spdk_conf
• Use config file to configure device, any device
• Used to evaluate performance of BDEV layer
![Page 8: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/8.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit
OCF - Introduction• What is OCF ?
• Open CAS Framework
• Cache Acceleration Software
• Selectively accelerates backend devices using more performant devices
• Why should I use it ?
• Fully Open Source and integrated into SPDK
• Proven Technology
• Higher Performance and Lower Latency
SPDK
BDEV Layer
VMVM
OCF bdev
NVMe bdev
Vhost Target
NVMe bdev
Open CAS Framework
Cache Device
Backend Device
![Page 9: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/9.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit
How it works? -- pass-Through1. Data is read from backend
device bypassing Cache
2. Data is written to backend device bypassing Cache
There are no caching benefits
Useful for testing
Application
OCF
Cache BdevBackend Bdev
Writes Reads
2
1
![Page 10: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/10.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit
How it works? -- Write-Through1. Data is read from backend
storage and copied to the cache device
2. All data is written synchronously to backend device and cache device
3. Subsequent reads of cached data are accelerated by cache device
Benefits repeated reads
Application
OCF
Cache BdevBackend Bdev
Writes Initial Read Subsequent Reads
2
1
3
“Accelerates re-reads of data that was read or written”
![Page 11: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/11.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit
How it works? -- Write-Back1. Data is read from backend
device and copied to the cache device
2. All writes go to the cache
3. Subsequent reads of cached data are accelerated by caching device
4. Dirty data is written opportunistically to backend device
Benefits reads & writes
Application
OCF
Cache BdevBackend Bdev
Writes Initial Read Subsequent Reads
2
1
3
4
“Accelerates both writes and re-reads”
![Page 12: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/12.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit 14
Exercise – Configure Your Bdev for OCF • vi examples/bdev/fio_plugin/bdev.conf.in
• [Malloc] Section for a ramdisk as caching device
• [Nvme] Section for a Nvme device as core device
• [OCF] Section for caching configuration
[Malloc]NumberOfLuns 1LunSizeInMB 128
[Nvme]TransportID "trtype:PCIe traddr:0000:06:00.0" Nvme0
[Split]Split Nvme0n1 2 102400
[OCF]OCF wb_test wb Malloc0 Nvme0n1p1
![Page 13: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/13.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit 15
Exercise – Edit FIO JOB• vi examples/bdev/fio_plugin/example_config.fio
[global]ioengine=./examples/bdev/fio_plugin/fio_pluginspdk_conf=./examples/bdev/fio_plugin/bdev.conf.inthread=1group_reporting=1direct=1verify=0time_based=1ramp_time=0runtime=2iodepth=128rw=randwritebs=4k[test]numjobs=1
![Page 14: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/14.jpg)
SPDK, PMDK & Vtune™ Amplifier Summit 16
Exercise – Start FIo• fio examples/bdev/fio_plugin/example_config.fio –filename=wb_test
Running Sample:[root@sm119 spdk]# fio examples/bdev/fio_plugin/example_config.fio --filename=wb_testtest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=128fio-3.3Starting 1 threadStarting SPDK v19.07 / DPDK 19.05.0 initialization...[ DPDK EAL parameters: fio --no-shconf -c 0x1 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid12819 ]EAL: No available hugepages reported in hugepages-1048576kBInserting cache wb_test
![Page 15: Zhou, Xueqiang (CAS AE, NSG) Hui, Chunyang (Software ... · SPDK, PMDK & Vtune™ Amplifier Summit How it works? -- Write-Back 1. Data is read from backend device and copied to the](https://reader036.fdocuments.us/reader036/viewer/2022063013/5fcbec6d54196d5bd12473c0/html5/thumbnails/15.jpg)