Virtual Machine Queue Driver Development

Sambhrama MundkurSr. Software Design EngineerCore Networkingsammun@microsoft.com

Agenda

• Overview of Virtual Machine Queue (VMQ)• VMQ capability advertisement• VM Queue allocation and deletion• Setting and clearing VMQ filters• Shared memory resources allocation• VMQ receive path• Embedded NIC switch • Diagnostics and Tools• Questions

VMQ Architecture

Parent PartitionVM1 VM2

Ethernet

VM BUS

TCP/IP TCP/IP

VM NIC 1 VM NIC 2

Network Virtual Service Provider

MiniportDriver

Switch/Routing unit

Q1 DefaultQueueQ2

RoutingVLAN filtering

Data Copy Port 1Port 2

NIC

Overview of Virtual Machine Queue

• Performance bottlenecks in receive processing in virtualized environment• Each incoming packet has to be examined to identify the

target virtual machine (VM)• Received data has to be copied from the parent partition’s

memory to the child partition’s memory address space• Lack of concurrency of interrupts and DPCs

Overview of Virtual Machine Queue (contd.)

• How does VMQ help?• Determine the target VM by MAC filtering in hardware• DMA received packets directly to the child partition’s memory

address space• Provide interrupt and DPC concurrency by indicating received

packets based on the target VM• Use switch capabilities of the NIC to provide VM to VM

loopback functionality

VMQ Capability Advertisement

• Receive filter capabilities advertised during miniport adapter initialization• Use standardized keywords to enable or disable VMQ

functionality

• Capabilities advertised• Enabled filters and queue types• Number of queues and filters• Supported queue properties• Lookahead-split support• NIC switch capabilities

Queue State Transitions

Queue Allocation

• Protocol driver issues an OID to allocate a queue and sets the parameters for the queue• Per-queue receive indication flag• Processor affinity• Queue name• Lookahead-split parameters

• NDIS generates a unique Queue ID and sets it before issuing the OID to the miniport driver

• Protocol driver can set filters after allocating the queue• Protocol driver issues an OID to complete the queue allocation

• Miniport driver can allocate shared memory and other resources

• Default queue (Queue ID 0) is always allocated

Queue Deletion

• Protocol driver clears all filters before freeing the queue• Protocol driver issues an OID to free the queue• Miniport driver

• Must immediately stop DMA to shared memory resources allocated for this queue

• Must generate status indication to indicate that the DMA has been stopped

• Waits for all outstanding NET_BUFFER_LIST (NBLs) for this queue to be returned

• Frees the allocated shared memory and hardware resources

• Default queue can never be freed

Updating Queue Parameters

• Protocol driver can issue an OID to update the queue parameters

• Queue parameters that can be updated are• Processor affinity• Queue name

VMQ Filter Operations

Setting and Clearing Filters

• Filters can be set and cleared any time after the queue has been allocated

• Protocol issues an OID to set the filter and the filter’s parameters• Queue ID• Array of header field test parameters. • For VMQ two header field tests can be set:

• Destination MAC address in the packet == specified MAC address

• VLAN ID in the packet == specified VLAN ID

• NDIS generates a unique filter ID and issues the OID to the miniport driver

Setting and Clearing Filters

• Miniport driver programs the hardware based on the filters• All header field tests of a particular filter must match to

indicate the packet to that queue• Multiple filters can be set on a queue• Packets must be indicated from the queue if any of the filters

pass

• Protocol driver can set filters on the default queue, too• Protocol driver issues an OID to clear a filter

• Provides the Filter ID of the filter

Shared Memory Resource Allocation

• Miniport driver calls the new DDI, NdisAllocateSharedMemory, to allocate shared memory resources for a queue• Initial allocation occurs when processing the queue allocation complete OID • Miniport driver can allocate shared memory as long as the queue has not been

freed

• Miniport driver specifies:• Queue ID• Length • Usage – Lookahead, post-Lookahead etc.

• NdisAllocateSharedMemory returns:• Virtual address of the allocated memory• Scatter/gather List• Shared memory handle (used in receive indications)• Allocation handle (used to free the memory)

Shared Memory Resources

• If lookahead-split is not enabled, specify that the memory will be used for receive buffer

• If lookahead-split has been enabled for the queue, allocate two shared memory buffers• Specify that usage is for the lookahead buffer• Specify that usage is for the post-lookahead (data) buffer

• Call NdisFreeSharedMemory to free the memory• Driver frees the resources when the queue is freed• Driver must wait for all NBLs using the shared memory to be

returned before calling this DDI

VMQ Packet Structures

VMQ Receive Path

• Indicate packets on a queue only if it passes all the filter field tests for a particular filter

• If per-queue receive indication flag is set: • NBLs from this queue must not be mixed with others• Must set ‘single queue indication’ flag in the Receive Flags

• Must set flag to indicate that the shared memory information is valid

• Out-of-band (OOB) information• Specify the Queue ID in packet OOB information• For VMQ, set the Filter ID in the OOB to zero

VMQ Receive Path

• Specify the SharedMemoryInfo in the NET_BUFFER• Set SharedMemoryHandle returned by NdisAllocateSharedMemory• Offset is the offset from the beginning of the shared memory buffer

• If lookahead split is enabled for the queue, each NET_BUFFER: • Must have two MDLs and corresponding SharedMemoryInfo structures• The post-lookahead buffer must have a backfill

• The protocol driver copies the contents of the lookahead buffer to the backfill

• Backfill must exist even if the packet is entirely in the lookahead buffer

• MDL Length and DataLength in the packet must be set as before

Queue Statistics

• NIC must maintain following per-queue statistics:• Number of transmitted and received packets• Number of transmitted and received bytes

• NIC can maintain per-packet type statistics• The statistics can be queried using the new statistics OID

• This is a mandatory Direct OID request• VMQ miniport driver must provide a Direct OID request

handler• Return statistics for a particular queue based on Queue ID• Default queue can be queried, too

Embedded NIC Switch

• Embedded switch on the NIC can assist in advanced features

• Advertise support during miniport adapter initialization• Loopback support

• Protocol driver issues an OID to enable these features• Loopback

• On transmit, Queue ID of the outgoing queue is set in packet OOB data

• NIC indicates the packet if it matches filters set on any other queue

• If no match is found, NIC transmits the packet on the wire

Diagnostics

• OIDs and WMI GUIDs • Enumerate the queues allocated on a miniport adapter• Query the parameters of a particular queue• Enumerate the filters allocated on a particular queue• Query the parameters of a particular filter• Handled by NDIS

• NDIS kernel debugger extensions• Performance counters exposed by NDIS

Call to Action

• Update drivers to NDIS 6.20 to take advantage of Windows 7 functionality

• Implement VMQ support in the miniport driver• Attend other related sessions in the DDC

Resources

• Windows 7 WDK will be available as of WinHEC• Windows Logo Program Web site:

http://www.microsoft.com/whdc/winlogo/default.mspx • NDIS 6 feedback alias:

ndis6fb@microsoft.com

Related Sessions

Session Day / Time

Network Power Management for Windows 7 Tues. 5:15-6:15

Windows Logo Program Tests for NDIS Mon. 11-12 andWed. 9:45-10:45

Discussion: Porting Drivers to NDIS 6.20 Wed. 8:30-9:30

Lab: How to Debug an NDIS 6.20 Driver Mon. 2:45-3:45 andWed. 9:45-10:45

Mobile Broadband Driver Development for Windows 7 Mon. 1:30-2:30

Virtual Machine Queue Architecture Review Tues. 2:45-3:45

Virtual Machine Queue Driver Development Tues. 4-5

Implementing Efficient RSS Capable Hardware and Drivers for Windows 7 Tues. 1:30-2:30

Developing TCP Chimney Drivers for Windows 7 Tues. 11-12

Driver Scalability Mon. 11-12 andTues. 11-12

Questions?