Neutron hybrid mode
description
Transcript of Neutron hybrid mode
SDN Architect, Nov 8 2013
Vinay Bannai
NEUTRON HYBRID MODE
Confidential and Proprietary2
PayPal offers flexible and innovative payment solutions for consumers and merchants of all sizes.
• 137 Million Active Users
• $300,000 Payments processed by PayPal each minute
• 193 markets / 26 currencies
• PayPal is the World’s Most Widely Used Digital Wallet
ABOUT PAYPAL
Confidential and Proprietary3
• Data Center Architecture
• Neutron Basics
• Overlays vs Physical Networks
• Use Cases
• Problem Definition
• Hybrid Solution
• Performance Data
• Analysis
• Q&A
INTRODUCTION
Confidential and Proprietary4
DATA CENTER ARCHITECTURE
Internet
Racks
Layer-3 switchAccess
Data Center
Layer-3 switchAggregation
Layer-3 routerCore
Bisection BW
Bisection BW
Bisection BW
Confidential and Proprietary5
NEW DATACENTER ARCHITECTURE
Internet
vswitches
Layer-3 switchAccess
Data Center
Layer-3 switchAggregation
Layer-3 routerCore
Bisection BW
Bisection BW
Bisection BW
VM VMVMVMVMVM VM VM VM VM
EdgeLayer
Confidential and Proprietary6
DATACENTER WITH VSWITCHES
Layer-3 switchAccess
Data Center
VM
VM
VM
VM
VM
Racks Racks Racks
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
Confidential and Proprietary7
NEUTRON BASICS
Confidential and Proprietary8
• Overlays provide connectivity between VMs and Network Devices using tunnels
• The physical core network does not need to be re-provisioned constantly
• The tunneling encap/decap is done at the edge in the virtual switch
• Decouples the tenant network address from the physical Data Center network address
• Easy to support overlapping address
• Tunneling techniques in vogue− VXLAN
− STT
− NVGRE
OVERLAY NETWORKS
Confidential and Proprietary9
• Physical Networks connect VM’s and Network Devices using provider network
• VM’s are first class citizens with the hypervisor and the networking devices
• No tunneling protocols used
• Tenant separation is achieved by using VLANs or IP subnetting
• Hard to achieve overlapping address spaces
• Underlying network needs to be provisioned with VLANs
PHYSICAL NETWORKS
Network Virtualization Layer
L2
VMVM VM VM VM
L2
L2
L3
VMVM VM VM VM
Tenant onOverlayNetwork
Tenant onPhysical Network
PHYSICAL VS OVERLAY
Confidential and Proprietary11
PROS & CONS
Function Hypervisor Bridged VMs (VLAN)
Tunneled VM’s
Throughput Best Better Worse
Latency Best Better Worse
Flexibility Worse Better Best
Overlapping IP addresses
Worse Worse Best
Operational Dependency
Worse Better Best
Confidential and Proprietary12
• Production Environment− Production website across multiple data centers
− Low latency and high throughput
− Bridged Mode
• Mergers & Acquisitions Private Community Cloud− Private Community Cloud
− Needs address isolation and overlapping
− Address isolation, Flexibility, low latency and high throughput
− Overlay Mode
• Development & QA Environment− Production development, QA & Staging
− Flexibility, high throughput but can tolerate higher latency
− Bridged and Overlay Mode
USE CASES
Confidential and Proprietary13
• Support flexibility, low latency, high throughput and overlapping address space all at the same time
• Support both bridged and overlay networks
• VM’s on a hypervisor should be able to choose networks
• Need a consistent deployment pattern
• Configurable by automation tools (puppet, chef, salt etc)
PROBLEM STATEMENT
Confidential and Proprietary14
TYPICAL VSWITCH
br-int
br-bond
VMTa
VMTb
VMTc
br-tun
Mgmt Interface
ProdInterface
VLAN 200
OverlayTraffic
BridgedTraffic
Bond Intf
IPInterface
HYBRID VSWITCH
Hypervisor
Confidential and Proprietary15
• Create the neutron networks− Flat Network
− neutron net-create bridged-flat --provider:network_type=flat --provider: physical_network=<Physnet>
− neutron subnet-create --allocation-pool start=10.x.x.100, end=10.x.x.200 bridged-flat --gateway 10.x.x.1 10.0.0.0/23 --name bridged-flat-subnet --enable_dhcp=False
− VLAN Network
− neutron net-create bridged-vlan --provider:network_type=vlan --provider: physical_network=<Physnet> --provider:segmentation_id=<vlan-id>
− neutron subnet-create --allocation-pool start=10.x.x.100, end=10.x.x.200 bridged-vlan 10.x.x.1 10.0.0.0/23 --name bridged-vlan-subnet
CONFIGURATION OF HYBRID MODE
Confidential and Proprietary16
• Neutron networks (contd.)− Overlay Network
− neutron net-create overylay-net
− neutron subnet-create --allocation-pool start=10.x.x.100, end=10.x.x.200 overlay-net --gateway 10.x.x.1 10.0.0.0/23 --name overlay-net-subnet
• On the compute node− Configure the bond
− ovs-vsctl add-br br-bond0
− Configure the OVS
− ovs-vsctl br-set-external-id br-bond0 bridgeid br-bond0
− ovs-vsctl set Bridge br-bond0 fail-mode=standalone
− ovs-vsctl add-port br-bond0 eth0 eth1
CONTD.
Confidential and Proprietary17
• To measure latency and throughput, we ran following tests
• Within a rack (L2 switching)− Bare metal to Bare metal
− Bridged VM to Bridged VM
− Tunneled VM to Tunneled VM
• Across racks (L3 switching)− Bare metal to Bare metal
− Bridged VM to Bridged VM
− tunneled VM to tunneled VM
• Across the Network Gateway − Bare metal to Bare metal (outside the cloud)
− Bridged VM to Bare metal (outside the cloud)
− tunneled VM to Bare metal (outside the cloud)
PERFORMANCE DATA
Confidential and Proprietary18
• Compute Hypervisors− 2 sockets, 16 cores/socket SandyBridge @ 2.6GHz (32 Hyper Threaded)
− 2 x 10G ports (Intel PCIe)
− RAM : 256GB
− Disk: 4 x 600GB in RAID-10
− RHEL 6.4 running OVS
• VM− vCPUs: 2
− RAM: 8GB
− Disk: 20GB
− RHEL 6.4
HYPERVISOR, VM AND OS DETAILS
Confidential and Proprietary19
TEST SETUP
X.X.X.X/23 Y.Y.Y.Y/23
Half rack withTwo Fault Zones
L3 GatewaysFor Overlays
X.X.X.X/23 Y.Y.Y.Y/23 X.X.X.X/23 Y.Y.Y.Y/23
Confidential and Proprietary20
• Tunneling VM uses STT (OVS)
• Bridged VM uses Flat Network (OVS)
• Used nttcp 1.47 for throughput
• Bi-directional TCP with varying buffer size
• Buffer size in bytes : [64,… 65536]
• MTU size : 1500 Bytes (on both bare metal and VM’s)
• Used ping for latency measurement (60 samples)
• Used python scripts and paramiko to run the tests
• Tests done with other traffic (Dev/QA)− Around 470+ active VM’s
− Around 100 Hypervisors
− Multiple half racks
TESTING METHODOLOGY
Confidential and Proprietary21
TEST SETUP FOR SAME RACK
Confidential and Proprietary22
WITHIN A RACK (L2 SWITCHING)THROUGHPUT
Confidential and Proprietary23
WITHIN A RACK (L2 SWITCHING)PING LATENCY
Confidential and Proprietary24
• Observations
• Results for buffer size < MTU size− Tunneled VM’s tend to have best overall throughput
− Bridged VM’s tend to better than bare metal
− OVS and tunnel optimizations at play
• Results for buffer size > MTU size− Tunneled VM’s and bare metal performance about the same
− Bridged VM’s bests both bare-metal and tunneled VMs (??)
• OVS and tunnel optimizations apply for buffer sizes smaller than MTU
• OVS optimization apply for buffer sizes greater than MTU
• Tunneled and Bridged VM’s have a slightly higher latency than bare metal
ANALYSIS
Confidential and Proprietary25
TEST SETUP ACROSS RACKS
Confidential and Proprietary26
ACROSS RACKS (L3 SWITCHING)THROUGHPUT
Confidential and Proprietary27
ACROSS R3ACKS (L SWITCHING)PING LATENCY
Confidential and Proprietary28
• No bridged VM’s in the tests (setup problem)
• Results for buffer size < MTU size− tunneled VM’s tend to have best overall throughput
− OVS and tunnel optimizations at play
• Results for buffer size > MTU size− tunneled VM’s and bare metal performance about the same
• OVS and tunnel optimizations apply for buffer sizes smaller than MTU
• tunneled and Bridged VM’s have a slightly higher latency than bare metal
ANALYSIS
Confidential and Proprietary29
TEST SETUP ACROSS L3 GATEWAY
Confidential and Proprietary30
ACROSS NETWORK GATEWAYTHROUGHPUT
Confidential and Proprietary31
ACROSS NETWORK GATEWAYPING LATENCY
Confidential and Proprietary32
• tunneled VM’s tend to have similar if not better throughput as bare metal or bridged VM
• tunneled VM’s have a slightly higher latency
• Bridged VM’s tend to have same overall throughput as the hypervisor
• Bridged VM’s tend to have same latency as the hypervisor
• Latency from a tunneled VM across L3 gateway is higher than Physical VMs due to extra hops, but need to re-run the tests
ANALYSIS
Confidential and Proprietary33
• Understand your network requirements− Latency, bandwidth throughput, flexibility
• Overlay Vs Physical
• Hybrid Mode
• Performance Analysis
• Make your deployment patterns simple and repeatable
• Future work− Additional performance tests
− VXLAN, NVGRE
− Varying MTU size
− Setup without background traffic
• Let me know if you are interested to collaborate
CONCLUSION & FUTURE WORK
THANK [email protected]