1

Virtual NVDIMM in Xen

Haozhong Zhang <haozhong.zhang@intel.com>

2

Outline

Introduction to NVDIMM

Overview of Virtual NVDIMM in Xen

Address Management

ACPI Emulation

Status & TODO

3

NVDIMM = Non-Volatile Dual In-line Memory Module

3

• Can be byte-addressable

• Persistent Storage

• retain data across power cycles

• Faster than SSD, higher density than RAM

44

CPU

RAM

NVDIMM

CPU

RAM NVDIMM

CPU

NVDIMMBLK Win

2-Level Memory

(2LM)

Persistent Memory

(pmem)

Persistent Block

(pblk)

✓

5

PMEM Architecture Overview

5

CPU

MC

RAM 0

RAM 1

SPA (System Physical Address):

/dev/pmemN

NVDIMM

Interleaveset

PMEM Namespace

ACPI NFIT

PMEM Region

PMEM Region

6

More about ACPI

6

CPU

MC

RAM 0

RAM 1 Label Storage Area

ACPI Namespace Device• Root “ACPI0012”

• _DSM: read/write label storage area

• _DSM: bad block scan (ARS)

• _DSM: vendor-specific

• _FIT: NVDIMM hotplug

• …

7

Cache Flush - ADR

7

CPU

MC

Cache

RAM 0

RAM 1

PMEM Region

PMEM Region

1. clwb/clflushopt/clflush

2. ADR (Asynchronous DRAM Refresh) automatically

8

Cache Flush – Flush Hint Structure

8

CPU

MC

Cache

RAM 0

RAM 1

PMEM Region

PMEM Region

1. clwb/clflushopt/clflush

2. Write to NFIT Flush Hint Structure

9

Outline

Introduction to NVDIMM

Overview of Virtual NVDIMM in Xen

Address Management

ACPI Emulation

Status & TODO

10

Usage Example of vNVDIMM in Xen

10

SPA

Hypervisor

Dom0

/dev/pmem0 /dev/pmem1

DAX file system

nvm1nvm2

/mnt/dax/

DomU

/dev/pmem0

DomU

/dev/pmem0

PMEM namespace 1 PMEM namespace 2

xl.cfg:

vnvdimm = [ ‘/dev/pmem0’ ]

xl.cfg:

vnvdimm = [ ‘/mnt/dax/nvm2’ ]

11

Architecture of vNVDIMM

11

vACPI

vDSM

vLabel

vACPI

MMIO

copy via hvmloader

QEMUxl:

• parse config

• get host pmem SPA

Linux Kernel (PMEM driver + DAX)

Dom0 DomU

Xen

Hypervisor

12

Outline

Introduction to NVDIMM

Overview of Virtual NVDIMM in Xen

Address Management

ACPI Emulation

Status & TODO

13

Detect Host PMEM Regions

13

• Dom0 PMEM Driver

• Parse NFIT and namespace labels

• Report PMEM SPA ranges to Xen hypervisor

• Xen Hypervisor

• Check overlap among PMEM SPA ranges and normal RAM regions

14

Guest Address Mapping

14

• Map as normal RAM

• EPT

• Fit into the existing memory management

• Reuse/Benefit from whatever already exists for normal memory management

• Room for Management Data Structures

• struct page_info per PMEM page

• Insufficient RAM space when PMEM is large

• Reserve and store on PMEM

15

Outline

Introduction to NVDIMM

Overview of Virtual NVDIMM in Xen

Address Management

ACPI Emulation

Status & TODO

16

hvmloader vs. QEMU

16

• Difficulties in Xen

• Not only dynamic data but also dynamic code in AML

• No runtime AML builder in Xen

• Conveniences in QEMU

• Runtime AML builder

• Existing NFIT and NVDIMM AML code implementation

17

Pass from QEMU

17

• Generate NFIT and AML code by QEMU

• Copy into DomU guest by QEMU

• Check and load by hvmloader

18

Collisions with hvmloader ACPI

18

• Duplicated tables

• Check table signature and reject duplicated tables (except SSDT)

• ACPI namespace device name collision

“NVDR”

Name (_HID, ”ACPI0012”)Method (_FIT) {…}Method (_DSM, …) {…}

+ Is “NVDR” already used?

Refuse to load

Device (NVDR) {Name (_HID, ”ACPI0012”)Method (_FIT) {…}Method (_DSM, …) {…}

}Yes

No

19

Outline

Introduction to NVDIMM

Overview of Virtual NVDIMM in Xen

Address Management

ACPI Emulation

Status & TODO

20

Status & TODO

20

• 2 RFC design documents

• https://lists.xen.org/archives/html/xen-devel/2016-02/msg00006.html

• https://lists.xen.org/archives/html/xen-devel/2016-07/msg01921.html

• Patches in development

• Xen / QEMU / Linux PMEM driver

• TODO

• NFIT Flush Hint Table

• RAS

• …

Q & A