NVDIMM block drivers with NFITMarch, 2016, SUSE Labs Taipei technology sharing day, Taipei

Joey LeeSUSE Labs Taipei

2

Agenda

• NVDIMM

• NFIT

• Conceptual model‒ nvdimm.txt

‒ Namespace/Region

‒ PMEM

‒ BLK

‒ NFIT

‒ BTT

• Q&A

NVDIMM

4

NVDIMM

• NVDIMM: Non-Volatile DIMM. Non-volatile memory in a DIMM form factor. [1]

http://www.enterprisetech.com/2014/02/20/micron-pushes-memory-roadmap-several-routes/

5

NVDIMM Architecture

6

NVDIMM Architecture - Block

7

SPA/DPA

• SPA: System Physical Address. A physical address on the host operating system. [1]

• DPA: DIMM Physical Address. An address within the memory in an NVDIMM. [1]

8

SPA/DPA

NVDIMM Namespace Specification, Intel

9

SPA/DPA Interleave

• DPA: DIMM Physical Address, is a DIMM-relative offset. With one DIMM in the system there would be a 1:1 system-physical-address:DPA association. Once more DIMMs are added an memory controller interleave must be decoded to determine the DPA associated with a given system-physical-address. BLK capacity always has a 1:1 relationship with a single-dimm's DPA range. [2]

10

SPA/DPA Interleave

NVDIMM Namespace Specification, Intel

11

PMEM

• Persistent Memory (or PMEM): Byte‐addressablememory which retains its contents across powerloss.A DIMM primarily containing non ‐volatile memory. [3]

• PMEM: A system physical address range where writes are persistent. A block device composed of PMEM is capable of DAX. A PMEM address range may span/interleave several DIMMs. [2]

• DAX: File system extensions to bypass the page cache and block layer to mmap persistent memory, from a PMEM block device, directly into a process address space. [2]

12

Block Window

• BW(Block Window): A set of registers consisting of a command register, a status register, and an aperture allowing the NVDIMM driver to read and write blocks of data to any persistent area on an NVDIMM. [1]

• BLK: A set of one or more programmable memory mapped apertures provided by a DIMM to access its media. This indirection precludes the performance benefit of interleaving, but enables DIMM-bounded failure modes. [2]

13

Block Window

14

BTT

• Block Translation Table. A software data structure, defined in the NVDIMM Namespace Specification, which prevents torn blocks when a write is interrupted by a system crash, hang, or power failure. [1]

• BTT: Existing software may have an expectation that the power-fail-atomicity of writes is at least one sector, 512 bytes. The BTT is an indirection table with atomic update semantics to front a PMEM/BLK block device driver and present arbitrary atomic sector sizes. [2]

15

Namespace

• NVDIMM Namespace: Similar to an NVMe Namespace or a Logical Unit (LUN) on a SCSI disk, this is a software mechanism for managing ranges of persistence on NVDIMMs. [1]

• NVDIMM Namespace Label: Labels, stored at a known location on NVDIMMs, that define the DIMM’s contribution to NVDIMM Namespaces. This is a software mechanism; the DIMM itself just sees the labels as part of the overall data stored on the DIMM. [1]

16

PMEM Namespace

NVDIMM Namespace Specification, Intel

17

BLK Namespace

NVDIMM Namespace Specification, Intel

18

Namespace

NVDIMM Namespace Specification, Intel

19

I/O on Multi-Range BLK Namspaces

NVDIMM Namespace Specification, Intel

20

Example Software Organization for Block I/O

NVDIMM Namespace Specification, Intel

21

Example Software Organization for Namespace Management

NVDIMM Namespace Specification, Intel

NFIT

23

NFIT

• NFIT: The NVDIMM Firmware Interface Table, which defines the ACPI-like information created by the BIOS to inform the OS about NVDIMMs in the system. [1]

• Device Specific Method (DSM): The NVDIMM root device and the NVDIMM devices can have device specific methods (_DSM) to provide additional functions specific to a particular NVDIMM implementation. [4]

24

NFIT Overview

ACPI 6.0 spec

25

NFIT

NVDIMM Block Window Driver Writer’s Guide, Intel

26

NFIT Information

NVDIMM Block Window Driver Writer’s Guide, Intel

27

NFIT SPA

NVDIMM Block Window Driver Writer’s Guide, Intel

28

NFIT Flush Hint

NVDIMM Block Window Driver Writer’s Guide, Intel

29

NFIT Block Window

NVDIMM Block Window Driver Writer’s Guide, Intel

30

NFIT (fixed)

NVDIMM Block Window Driver Writer’s Guide, Intel

Conceptual model

32

Conceptual model of nvdimm.txt

33

Conceptual model of driver/nvdimm/namsepace, region

34

Conceptual model of driver/nvdimm/pmem

35

Conceptual model of driver/nvdimm/blk

36

Conceptual model of driver/nvdimm/btt

37

Conceptual model of driver/acpi/nfit

Q&A

39

Reference

• [1] NVDIMM Block Window Driver Writer’s Guide, Intel

• [2] Documentation/nvdimm/nvdimm.txt, Linux Kernel

• [3] NVDIMM Namespace Specification, Intel

• [4] NVDIMM DSM Interface Example, Intel

Thank you.

40

Feedback tojlee@suse.com

Corporate HeadquartersMaxfeldstrasse 590409 NurembergGermany

+49 911 740 53 0 (Worldwide)www.suse.com

Join us on:www.opensuse.org

42

Unpublished Work of SUSE. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.