Sogang University Distributed Computing & Communication Lab.

Concurrent Direct Network Access for Virtual Machine MonitorsPaul Willmann, Jeffrey Shafer, David Carr, Aravind Menon,

Scott Rixner, Alan L. Cox, Willy Zwaenepoel

Kwon-yong LeeKwon-yong Lee

Distributed Computing & Communication Lab.Distributed Computing & Communication Lab.(URL: http://dcclab.sogang.ac.kr)(URL: http://dcclab.sogang.ac.kr)

Dept. of Computer Science Dept. of Computer Science Sogang UniversitySogang University

Seoul, KoreaSeoul, Korea

Tel : +82-2-3273-8783Tel : +82-2-3273-8783 Email : dlrnjsdyd@sogang.ac.krEmail : dlrnjsdyd@sogang.ac.kr

2

Necessity of new I/O Virtualization Tech.

Improving the efficiency of VMMs Hardware support for virtualization Reducing the software overhead of virtualization

Networking by VMM Virtual Network Interface

• Multiplexed in software onto a physical network interface card (NIC)

Overhead of the software-based network virtualization• Limiting network performance

No clear solution to improve the efficiency of I/O virtualization

3

Xen

Key functions in order to provide VM environments Allocating the physical resources of the machine to the

guest OSes and isolating them from each other

Receiving all interrupts in the system and passing them on to the guest OSes, as appropriate

All I/O operations go through Xen.• In order to ensure fair and non-overlapping access to I/O

devices by the guests

PerformanceSystem Transmit (Mb/s) Receive (Mb/s)

Native Linux 5126 3629

Xen 3.0 Guest 1602 1112

Organization of the Xen VMM

4

Networking in Xen

HypervisorControl

Hypervisor

PageFlippin

gPacketData

Interrupt Dispatch

NIC

Driver DomainBack-End

DriverEtherne

tBridge

NIC Driver

Back-End

Driver

GuestDomain 1

Front-End

Driver

GuestDomain 2

Front-End

Driver

CPU / Memory / Disk / Other Devices

DriverContro

lPacketData Control +

DataInterrupts

5

Hypervisor and Driver Domain Operation

Hypervisor

PageFlippin

gPacketData

Interrupt Dispatch

NIC

Driver DomainBack-End

DriverEtherne

tBridge

NIC Driver

Back-End

Driver

GuestDomain 1

Front-End

Driver

GuestDomain 2

Front-End

Driver

CPU / Memory / Disk / Other Devices

DriverContro

lPacketData

Packet Transmit

6

Hypervisor and Driver Domain Operation

Hypervisor

PageFlippin

gPacketData

Interrupt Dispatch

NIC

Driver DomainBack-End

DriverEtherne

tBridge

NIC Driver

Back-End

Driver

GuestDomain 1

Front-End

Driver

GuestDomain 2

Front-End

Driver

CPU / Memory / Disk / Other Devices

DriverContro

lPacketData Interrupts

Packet Receive

Virtual Interrupts

Virtual Interrupts

7

Device Driver Operation

Interactions between the device driver and the NIC Programmed I/O (PIO) operations from the driver to the

NIC Direct memory access (DMA) transfers by the NIC to

read or write host memory Physical interrupts from the NIC to invoke the device

driver

The device driver directs the NIC To send packets from buffers in host memory To place received packets into pre-allocated buffers in

host memory

8

Device Driver Operation

Host OS

NIC

mailbox

DeviceDriver Buffer

PIO

Transmit Receive

DMA descriptor ring

Producer Index

DMA descriptor

(Buffer’s lengthand

physical address)

Newpacke

t

NewDMA

descriptor

Packet Transmit

A new descriptor is available.(ring index)

DMA read

Consumer Index

With CDNA, each guest operating system is connected directly to its own network interface.→ Elimination many of the overheads of network virtualization Communication overheads between the guest and driver domains Software multiplexing overheads within the driver domain

9

Concurrent Direct Network Access

10

Concurrent Direct Network Access

CDNA Network Interface Supporting multiple contexts in hardware

• Acts as if it is an independent physical network interface• Controlled by a separate device driver instance

Each guest can transmit and receive network traffic using its own private context directly.

Network interface Multiplexing the traffic across all of its active contexts

Hypervisor Providing protection across the contexts

11

Concurrent Direct Network Access

Multiplexing network traffic on the NIC Network interface

• Identifying the source/target OS for all network traffic• Providing independent hardware contexts and associating a

unique Ethernet MAC address with each context• Each context must include a unique set of mailboxes.

– This isolates the activity of each guest OS, so that the NIC can distinguish between the different guests.

Hypervisor• Assigning a unique hardware context on the NIC to each guest

OS– By simply mapping I/O locations for that context’s mailboxes into

the guest’s address space

Device driver within the guest OS• Interact with its context exactly

– Creating DMA descriptors and Updating a mailbox on the NIC via PIO

12

Concurrent Direct Network Access

Interrupt Delivery The hardware contexts on the NIC must be able to

interrupt their respective guests.

NIC• Encoding the interrupt set of contexts in an interrupt bit

vector• Transferring into the hypervisor’s memory space using

DMA• Raising a physical interrupt, which invokes the hypervisor’s

interrupt service routine

Hypervisor• Decoding all of the pending interrupt bit vectors• Scheduling virtual interrupts to each of the guest OSes

13

Concurrent Direct Network Access

DMA Memory Protection Key I/O protection violation

• Transmit packets : Security hole• Receive packets : Corrupting memory that is in use

Additional security features are needed. The CDNA architecture for the isolation between guests

• Validating and protecting all DMA descriptors • Ensuring sources or targets of DMA accesses

Protection provided by CDNA• The device driver in each guest must call into the

hypervisor to perform the en-queue operation.• Hypervisor’s exclusive write access to the host memory

region– Preventing from independently en-queuing unauthorized DMA

descriptors

14

CDNA NIC Implementation

RiceNIC Programmable and reconfigurable FPGA-based Gigabit

Ethernet network interface

Additional mailbox storage and handling logic

Modification• To provide multiple protected contexts• To multiplex network traffic• To interact with the hypervisor through a dedicated context

to allow privileged management operations

15

Evaluation

Xen 3.0 Unstable

NIC A single dual-port Intel Pro/1000 MT NIC Two RiceNICs configured to support CDNA

16

Evaluation – Single Guest Performance

17

Evaluation – Single Guest Performance

CDNA I/O virtualization architecture provides significant performance improvements over Xen for both transmit and receive. Transmit side

• CDNA requires half the processor resource to deliver about 200 Mb/s higher throughput.

Receive side• CDNA requires 60% of the processor resources to deliver

about 750 Mb/s higher throughput

18

Evaluation – Memory Protection

The software-based protection mechanisms in CDNA can potentially be replaced by a hardware IOMMU.

19

Evaluation – Scalability

Summary

To support CDNA, A VMM would only need to add mechanisms To deliver interrupts as directed by the network

interface To perform DMA memory protection

Required to enable a commodity NIC to support CDNA The NIC must provide multiple contexts that can be

accessed by programmed I/O. The NIC must support several mailboxes within each

context.

20