NVMe Virtualization ND - SNIA · NVMe virtualization approaches r Implementing SCSI to NVMe...

NVMe virtualization ideas for Machines on Cloud

Sangeeth KeeriyadathIBM

k.sangeeth@in.ibm.com

@ksangeek

day2dayunix.sangeek.com

Disclaimer

All of the contents of this presentation are based on my understanding of NVM Express® and my academic interests to explore the possibilities of virtualizing NVMe. The contents in this presentation do not necessarily represent IBM’s positions, strategies or opinions.

Topics

r NVMe primerr Past storage virtualization learningsr Virtualization considerationsr NVMe virtualization approaches

rBlind Mode : SCSI to NVMe translationrVirtual Mode : pure NVMe virtual stackrPhysical Mode : SR-IOV

r NVMe over Fabrics virtualizationr Other deliberations

NVMe primer [1] [2]

NVM Express® (NVMe) is an inherently parallel and high-performinginterface and command set designed for non-volatile memory based storager The interface provides optimized command submission and

completion paths resulting in :r Lower latencyr Increased bandwidth

r Simple command setr Support for parallel operation ( no lock contention ) by supporting up

to 65,535 I/O Queues with up to 64K outstanding commands per I/O Queue

r Suitable for complementing the benefits of multi-core CPU systems application parallelism - Command Submission/Completion queue ( SQ/CQ ) bound to core, process or thread

NVMe SSD

NVMe view [3]

NVM Express Controller

Namespace A(NSID 1)

Namespace B(NSID 2)

Namespace C(NSID 3)

Controller ManagementAdmin Submission / Completion Queues

I/OSubmissionQueue A

I/OSubmissionQueue X

I/OSubmissionQueue Y

I/OCompletionQueue A

I/OCompletionQueue Z

NVMe benefitsr Storage stack reductionr Designed considering next generation NVM

based devicesr Bind interrupts to CPUsr Interconnect choices :

rNVMe over PCIerNVMe over Fabric (RDMA, Fibre Channel)

Why virtualize ?

r Key component of cloud computingr Optimizes utilization of physical resourcesr Improves storage utilizationr Simplified storage management - provides a

simple and consistent interface to complex functions

r Easier high availability(HA) solutions

Learnings : IBM PowerVM success storyIBM® PowerVM® provides the industrial-strength virtualization solution for IBM Power Systems™ servers and blades that run IBM AIX®, IBM i and Linux workloads.Virtual I/O Server(VIOS) facilitates the sharing of physical I/O resources among guest Virtual Machines(VM). [4]

r NPIV [5]N-Port ID Virtualization(NPIV) is a T11 standard technology for virtualization of Fibre Channel networks. Enables connection of multiple VMs to one physical port of a Fibre Channel adapter. VIOS facilitates storage adapter sharing.

r vSCSI [6]Using virtual SCSI, VM can share disk storage, tape or optical devices that are assigned to the VIOS. VIOS does the storage virtualization, performs SCSI emulation and acts as SCSI target. 8

PowerVM storage virtualization view

Virtual IO Server

Power Hypervisor

vSCSI VM

A B C D

E F HG

vFC VM

vFC_CvS_CvS_SvFC_S

G H H DG C

Why virtualize NVMe ?

r Limited PCIe slots and need to share the benefits amongst VMs ( high VM density )

r Use cases :rServer side caching of VM datarVM boot disk

NVMe being highly scalable is well suited for various virtualization exploitations

NVMe virtualization approaches

r Implementing SCSI to NVMe translation layer on the Hypervisor. ( Blind Mode ).

r Pure virtual NVMe stack by distributing I/O queues amongst hosted VMs. ( Virtual Mode ).

r SR-IOV based NVMe controllers per virtual functions ( Physical mode ).

Blind Mode : SCSI to NVMe translationr Motivation

r Lot of storage stack ecosystem built around SCSI architecture model, protocol and interfaces

r Preserve software infrastructure investmentsr Method

r Implement a “SCSI to NVMe translator” layer built logically below the operating system SCSI storage stack and above the NVM Express driver

r Detailed in “NVM Express: SCSI Translation Reference” document [7]r SCSI primary/block commands to NVMe command mappingr Common SCSI Field translations e.g. “PRODUCT IDENTIFICATION”

field would be translated to first 16 bytes of the Model Number (MN) field within the “Identify Controller Data”

r NVMe “Status Code” to SCSI “Status Code, Sense Key, Additional Sense Key” 12

OS Storage stack : SCSI to NVMe translated

NVMe SSD

File System

Logical Volume Manager

SCSI Block disk driver

SCSI to NVMe translator

NVM Express Driver

Application

Success Completion

Invalid Command Opcode

GOOD / NO SENSE

CHECK CONDITION / ILLEGAL REQUEST / INVALID COMMAND OPERATION CODE

Identify command

INQUIRYREPORT LUNSREAD CAPACITY

READ(6), READ(10),READ(12), READ(16)

Blind Mode View : NVMe unaware VM

VIOSVM A

NVMe SSD

r SCSI to NVMe translation performed on the storage hypervisorr Virtual Machine(VM)’s Operating System(OS) storage stack is

unmodifiedr Performance consideration : “storage hypervisor” could use a “hardware

accelerated SCSI to NVMe translator”

TranslatorvSCSI HOST

Block disk driver

SCSI-NVMe translator

NVMe driver

Model suitable for server side storage virtualization solutions and preserve NVMe benefits

Virtual Mode : pure NVMe virtual stack

r Motivation

r Share NVM storage capacity amongst Virtual Machines(VMs)r VMs with pure NVMe stack retain the various NVMe performance

characteristics :r Low latencyr Parallelism

r Easier on-demand scaling(grow/shrink) implementationr Suitable for future “NVMe over Fabrics” implementations in data-

centers

Virtual Mode : pure NVMe virtual stack

r Method

To be disclosed during presentation

Virtual Mode View : pure NVMe stack on VM

To be disclosed during presentation

Physical Mode : SR-IOV [8]

r Motivationr Single Root I/O Virtualization (SR-IOV) is a specification that allows a PCIe

device to appear as multiple separate physical PCIe devicesr Inherent QoS capabilities and configuration

r Methodr Partition adapter capability logically into multiple separate PCI function

called “virtual functions”r Each “virtual function” could be individually assigned to a Virtual

Machine(VM)

NVM Express Controller

NVMe SR-IOV view

19NVMe SSD

Physical Function Virtual Function( VF A )

Virtual Function( VF B )

I/OQueues

AdminQueues

NVMe driver

Hypervisor

I/OQueues

AdminQueues

NVMe driver

VM A VM B

NVMe over Fabrics

r Extends NVMe onto data center fabrics such as Ethernet, Fibre Channel and InfiniBandTM

r Distance connectivity to storage systems with NVMe devicesr Scaling up of the NVMe devices in large solutions

r Two types of fabric transports for NVMe are currently under development: r NVMe over Fabrics using RDMA ( InfiniBand, RoCE and iWARP )

r RDMA verbsr NVMe over Fabrics using Fibre Channel (FC-NVMe)

r Backward compatible with Fibre Channel Protocol (FCP) transporting SCSI

NVMe over fabrics view

NVMe SSD

Virtual Machine A InfinibandTM

InfinibandTM

Infinibandfabric

Ethernetfabric

Fibre Channel fabric

Server NVMe Storage box

Virtual Machine B

Virtual Machine C

NVMe over Fabrics virtualization

r NVMe over Fabrics using FC ( NPIV ) [9]

r NVMe over Fabrics using RDMA

Other deliberations

r SSD endurance – Drive Writes Per Day (DWPD) r Virtual Machine migrationr Interrupt driven model or polling

Referencesr [1] NVM ExpressTM Infrastructure - Exploring Data Center PCIe® Topologies :

http://www.nvmexpress.org/wp-content/uploads/NVMe_Infrastructure_final1.pdfr [2] A Comparison of NVMe and AHCI “Don H Walker” :

https://sata-io.org/system/files/member-downloads/NVMe%20and%20AHCI_%20_long_.pdfr [3] NVM Express Overview

http://www.nvmexpress.org/nvm-express-overview/r [4] IBM PowerVM Virtual I/O Server overview :

http://www.ibm.com/support/knowledgecenter/POWER8/p8hb1/p8hb1_vios_virtualioserveroverview.htmr [5] IBM PowerVM “Virtual Fibre Channel” :

http://www.ibm.com/support/knowledgecenter/POWER8/p8hb1/p8hat_vfc.htmr [6] IBM PowerVM “Virtual SCSI” :

http://www.ibm.com/support/knowledgecenter/POWER8/p8hb1/p8hb1_vios_concepts_stor.htmr [7] NVM Express: SCSI Translation Reference :

http://www.nvmexpress.org/wp-content/uploads/NVM-Express-SCSI-Translation-Reference-1_1-Gold.pdfr [8] I/O Virtualization in Enterprise SSDs :

http://www.snia.org/sites/default/files/SDC15_presentations/virt/ZhiminDing_IO_Virtualization_eSSD.pdfr [9] FC-NVMe NVMe over Fabrics “QLOGIC” :

http://www.qlogic.com/Resources/Documents/WhitePapers/Adapters/WP_FC-NVMe.pdf

Credits [ IBM Systems ]

r Hemanta Dutta, AIX Storage & IO SWr Mallesh Lepakshaiah, Virtual IO Serverr Ninad Palsule, AIX Virtual Server Storager Sudhir Maddali, AIX Storage Device Driversr Venkata Anumula, AIX Storage Device Drivers

Thank You

NVMe Virtualization ND - SNIA · NVMe virtualization approaches r Implementing SCSI to NVMe...

Documents

Transcript of NVMe Virtualization ND - SNIA · NVMe virtualization approaches r Implementing SCSI to NVMe...

Educational Infrastructure Using Virtualization ... · Educational Infrastructure Using Virtualization ... Using server virtualization, the hypervisor isolates each server’s Fig.

Hypervisor, Virtualization Stack, And Device Virtualization Architectures Mikael Nyström – TrueSec AB MVP Windows Server – Setup/Deployment.

Power E ciency of Hypervisor-based Virtualization versus Container-based Virtualization · 2016-08-30 · Power E ciency of Hypervisor-based Virtualization versus Container-based

Virtualization in Multilevel Security Environments...Virtualization Technologies • Type 1 - Hypervisor-based virtualization > xVM - style (think XEN) > Logical Domains (LDOM) - firmware-style,

Security Recommendations for Server-based Hypervisor Platforms€¦ · This revision includes additional technologies for device virtualization such as para-virtualization, passthrough

A Hypervisor IPS based on Hardware Assisted Virtualization Technology

Windows Server Virtualization & The Windows Hypervisor Background

Step-By-Step Build and Deploy Xen Hypervisor Virtualization on Linux OpenSUSE 11.3 Server

Server virtualization - ETH Z · PDF filevirtual NIC Server virtualization Hypervisor VM1 VM 2 VM N pNIC Physical NIC vNIC vNIC vNIC

MDev-NVMe: A NVMe Storage Virtualization Solution with … · been designed from a logical device interface where stor-age media is attached via a PCIe bus [26]. NVMe SSDs can deliver

Virtualization on the Mid-Range · Virtualization on the Mid-Range Jaqui Lynch Mainline Information Systems ... Math 101 and Consolidation ... POWER Virtualization or POWER Hypervisor

Resource Over-Commitment for KVM Virtualization Environments · Virtualization over-commitment overview • Virtualization over-commitment is using the virtualization technology (hypervisor)

A Performance Evaluation of Hypervisor, Unikernel, …...A Performance Evaluation of Hypervisor, Unikernel, and Container Network I/O Virtualization Pekka Enberg Master’s Thesis

Beyond the Hypervisor - SCALE 18x · 2014-02-23 · Beyond the Hypervisor - Into the Processor Virtualization Functions Pulled Into Hardware Over the Years: –Shadow VMCS - tracking

Hypervisor, Virtualization Stack, And Device Virtualization Architectures Mike Neil Product Unit Manager Windows Virtualization Microsoft Corporation.

A Performance Evaluation of Hypervisor, Unikernel, and ... · A Performance Evaluation of Hypervisor, Unikernel, and Container Network I/O Virtualization Pekka Enberg Master’s Thesis

Datasheet of Windows Barebone series - QNAPdownload.qnap.com/station/virtualization/Virtualization-Station_EN... · -Critrix XenServer-Microsoft Hyper-V Hypervisor (Hosted)-Oracle

NVMe Virtualization - SNIA · •NVMe Virtualization Performance. What is NVMe •NVM Express is a standardized high performance software ... AIX WPARs Linux Containers Dockers BSD

Architectural Support for Hypervisor-Secure Virtualizationpalms.ee.princeton.edu/system/files/asplos2012_wfooter.pdfArchitectural Support for Hypervisor-Secure Virtualization Jakub

Oracle Solaris 11 Network Virtualization and Network Resource · Hypervisor Virtualization..... 4! Architecture ... Network resource management is another important aspect of the