NSDI April , Andromeda - USENIX · Andromeda Data Plane VM Host Management Plane Open vSwitch...
Transcript of NSDI April , Andromeda - USENIX · Andromeda Data Plane VM Host Management Plane Open vSwitch...
-
AndromedaPerformance, Isolation, and Velocit at
Scale in Cloud Net ork Virtualization
NSDIApril ,
-
Andromeda Goals
Performance and IsolationHigh throughput and lo latenc , regardless of the actions of other tenants
VelocitQuickl de elop and deplo ne features and performance impro ements
ScalabilitLarge net orks, man tenants, rapid pro isioning
-
Cluster xx10.1.0.0/16
Cluster yy10.2.0.0/16
vmA
vmX
vmY
vmB
vmC
vmD
vmE
vmZ
vmV
virtual switch
vmF
vmM
vmP
vmN
vmL
vmQ
virtual switch
virtual switch
virtual switch
virtual switch
Host 10.1.1.3
Host 10.1.2.4
Host 10.1.2.5
Host 10.2.1.7
Host 10.1.1.9
Virtual IP192.168.0.2192.168.0.3192.168.0.4192.168.0.5192.168.0.6192.168.0.710.240.0.310.240.0.610.240.0.7
vnid111111222
Host:keylocal:1710.1.1.3:110.1.2.4:110.1.2.4:210.1.2.4:310.2.1.7:110.1.1.3:210.1.1.3:210.1.1.
Net ork Virtualization
-
Andromeda ArchitectureVM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM ControllerVM ControllerVM Controller VM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM Host
Virtual s itch
VM
VM VM
RPC
E tended OpenFlo
Match Actions Match Actions Match Actions
-
Andromeda ArchitectureVM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM ControllerVM ControllerVM Controller VM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM Host
VM
Virtual s itch
VM
VM VM
RPC
E tended OpenFlo
Match Actions Match Actions Match Actions
1. New VM added
-
Andromeda ArchitectureVM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM ControllerVM ControllerVM Controller VM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM Host
VM
Virtual s itch
VM
VM VM
RPC
E tended OpenFlo
Match Actions Match Actions Match Actions
2. Install flows from other VMs to the new VM.
1. New VM added
-
Andromeda ArchitectureVM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM ControllerVM ControllerVM Controller VM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM Host
VM
Virtual s itch
VM
VM VM
RPC
E tended OpenFlo
Match Actions Match Actions Match Actions
3. Install flows from the new VM to other VMs in the network.
2. Install flows from other VMs to the new VM.
1. New VM added
-
Scaling Goals
Global connecti it
Large irtual net orks k+ VMs
Rapid pro isioningEnable on-demand orkloads
-
Programming Time for Large Net orksSetup:❖ VMs are placed on , hosts❖ VM Controller partitions
Programming time is O n×H n = number of VMs H = number of hosts
Quadratic scaling leads to pro isioning challenges❖ Control plane CPU and memor❖ Dataplane memor
-
mA
mX
mY
mB
mD
irtual s itch
mC
irtual s itch
Host . . .
Host . . .
Scaling ith Ho erboards
HoverboardHoverboardHoverboard
vmE
vmZ
vmV
virtual switch
Host 10.1.2.5
low priority route
-
mA
mX
mY
mB
mD
irtual s itch
mC
irtual s itch
Host . . .
Host . . .
Ho erboard Offloading
HoverboardHoverboardHoverboard
vmE
vmZ
vmV
virtual switch
Host 10.1.2.5
vmX → vmZ offload flow
low priority route
-
mA
mX
mY
mB
mD
irtual s itch
mC
irtual s itch
Host . . .
Host . . .
Ho erboard Offloading
HoverboardHoverboardHoverboard
vmE
vmZ
vmV
virtual switch
Host 10.1.2.5
low priority route
vmX → vmZ offload flow
VM Controller
OpenFlow Front Endstats
flow prog
ramming
-
Ho erboards reduce time to program net ork connecti it for large net orks
❖ ✕ faster for a , -VM net ork
Programming Time for Large Net orks
-
Wh Ho erboards Are Effecti e
% of VM pairs ha e peak throughput < kbps
o er % of VM pairs ne er communicate
Peak throughput for all VM pairs in all virtual networks in one cluster over a 30-minute interval
Today, more than 99.5% of traffic is offloaded.
-
Andromeda Data Plane
VM Host Open vSwitch
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance
Userspace dataplane, live migration, and hitless upgrades for feature velocity
Manages on-host Flow Tables
-
Andromeda Data Plane
VM Host Management Plane
Open vSwitch
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance
Userspace dataplane, live migration, and hitless upgrades for feature velocity
Manages on-host VMs
-
Andromeda Data Plane
VM Host Management Plane
Open vSwitch
Andromeda Fast Path
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance
Userspace dataplane, live migration, and hitless upgrades for feature velocity
Busy polls physical & virtual NIC queues, forwards VM packets
-
Andromeda Data Plane
VM Host Management Plane
Open vSwitch
Andromeda Fast PathMatch Action
Flo cache
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance
Userspace dataplane, live migration, and hitless upgrades for feature velocity
Routes packet, applies per-flow Fast Path actions (encap, decap, etc)
-
Andromeda Data Plane
VM Host Management Plane
Open vSwitch
Guest VM Coprocessor
Andromeda Fast PathMatch Action
Flo cache
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance
Userspace dataplane, live migration, and hitless upgrades for feature velocity
Per-VM attributed threads for executing CPU-intensive packet ops (e.g., DoS)
-
Andromeda Data Plane
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast PathMatch Action
Flo cache
miss insert
Host OS Kernel
NIC
shared memory ring
Extended OpenFlow
OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance
Userspace dataplane, live migration, and hitless upgrades for feature velocity
Fast Path polls guest VM rings & copies packets to/from guest VM memory
-
Data Plane - Fast Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End High performance traffic processed end-to-end on Fast Path
> 30Gb/s throughput & > 3M pps on one core
Flow Table performs routing, encap/decap, etc.
Fast Path polls virtual & physical NIC rings
PacketMatch Action
Flo cache
Pull packet from NICParse, TcpDump, ...
-
Data Plane - Fast Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End High performance traffic processed end-to-end on Fast Path
> 30Gb/s throughput & > 3M pps on one core
Flow Table performs routing, encap/decap, etc.
Fast Path polls virtual & physical NIC rings
Packet
Match Action
Flow LookupRoute, decap, ...
-
Data Plane - Fast Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End High performance traffic processed end-to-end on Fast Path
> 30Gb/s throughput & > 3M pps on one core
Flow Table performs routing, encap/decap, etc.
Fast Path polls virtual & physical NIC rings
Packet
Match Action
Deliver packet to VMCopy, update rings, ...
Flo cache
-
Data Plane - Coprocessor Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End
PacketMatch Action
Flo cache
Pull packet from NICParse, TcpDump, ...
Coprocessors are per-VM threads CPU attributed to VM container
Coprocessors execute CPU-intensive packet ops such as DoS
Decouples feature growth from Fast Path speed
-
Data Plane - Coprocessor Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End
Packet
Match Action
Flow LookupRoute, decap, set Coprocessor stages...
Coprocessors are per-VM threads CPU attributed to VM container
Coprocessors execute CPU-intensive packet ops such as DoS
Decouples feature growth from Fast Path speed
-
Data Plane - Coprocessor Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End
Match Action
Flo cache
Packet
Send to CoprocessorApply Coprocessor stages (e.g., DoS)
Coprocessors are per-VM threads CPU attributed to VM container
Coprocessors execute CPU-intensive packet ops such as DoS
Decouples feature growth from Fast Path speed
-
Data Plane - Coprocessor Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End
Packet
Match Action
Flo cache
Deliver packet to VMCopy, update rings, ...
Coprocessors are per-VM threads CPU attributed to VM container
Coprocessors execute CPU-intensive packet ops such as DoS
Decouples feature growth from Fast Path speed
-
VM-VM Throughput Single core per host for dataplane Fast Path. Sk lake testbed hosts.
Both hosts connected to same Top of Rack s itch.
-
VM-VM Round Trip Latenc Single core per host for dataplane Fast Path. Sk lake testbed hosts.
Both hosts connected to same Top of Rack s itch.
-
CPU Efficienc
Minimizing host and guest net ork CPU c cles per b te CPB is critical
Since initial production release, e ha e impro ed CPB b > 6 as measured on sender + recei er host during a multi-stream benchmark.
Andromeda . + use a single core per host for the dataplane Fast Path. Results from Sand bridge testbed hosts connected to same ToR s itch.
-
CPU Efficienc E olution
Host: 43.5Guest:16.0
Host: 30.4Guest:12.3
Host: 5.4Guest: 5.6
Host: 2.6Guest: 5.0
Host: 2.0Guest: 4.9
Andromeda 1.0Kernel datapath
Andromeda 1.5Optimize pipeline
Andromeda 2.0OS bypass, 1 thread hop
Andromeda 2.1Remove thread hop
Andromeda 2.2Memory copy offload
31
-
VelocitA rapid release c cle enables s ift deplo ment of features & bug fi es.
Our dataplane has eekl rollouts ia non-disrupti e upgrades.
Li e migration allo s VMs to be migrated bet een ph sical host ithout disruption, enabling transparent host maintenance.
-
Dataplane Hitless Upgrade /
Physical NIC
Upgrade Brownout
Old Dataplane state is transferred to New Dataplane in the background
Old Dataplane continues serving physical NIC & virtual NIC queues
State XferOld Dataplane New Dataplane
Guest VM
-
Dataplane Hitless Upgrade /
Physical NIC
State XferOld Dataplane New Dataplane
Guest VM
Upgrade Blackout
Old Dataplane stops serving virtual & physical NIC queues
Then, any updated (delta) Old Dataplane state is transferred to New Dataplane
-
Dataplane Hitless Upgrade /
Physical NIC
New Dataplane
Guest VM
Upgrade Complete
State xfer done. Median blackout time is 270ms.
New Dataplane starts serving VM virtual NIC & physical NIC queues
Old dataplane terminated
-
ConclusionWe ha e discussed the design and e olution of Andromeda
Control plane scalabilit & Rapid pro isioning
● Ho erboard model a oids programming long tail of mostl idle flo s on VM host. Scales to k VMs/net ork
High performance & Feature elocit
● OS B pass dedicated CPU dataplane pro ides high performance > Gb/s, > M pps ith core & eekl non-disrupti e updates