Open stack networking_101_update_2014-os-meetups

© 2014 VMware Inc. All rights reserved. © 2014 VMware Inc. All rights reserved.

12/11/2014

OpenStack Networking Neutron 101 + Updates of 2014 Yves Fauser Solution Architect @ VMware

Agenda

•  Network Models •  Nova-Networking vs. Neutron refresher

–  Nova-Networking quick overview –  Nova-Networking Multi-Host mode –  Nova-Networking vs. Neutron at a glance

•  Neutron plugin concept refresher

•  Service plugins •  ML2 plugin vs. monolithic Plugins

•  New in Juno –  Distributed Virtual Router for OVS mechanism driver –  Neutron L3 High-Availability for virtual routers –  Neutron IPv6 Support

OpenStack Networking – Flat •  In the simple ‘flat’ networking model, all instances (VMs) are bridged to a physical adapter

•  L3 first-hop routing is either provided by the physical networking devices (flat model), or by OpenStack L3 Service (flat-DHCP model)

•  Sufficient in single tenant or ‘full trust’ use cases were no segmentation is needed (beside iptables/ebtables between VM interfaces and bridge)

•  Doesn’t provide multi-tenancy, L2 isolation and overlapping IP address support

•  Available in Neutron and in Nova-Networking

VM VM VM VM

VM VM VM VM

VM VM VM VM

VM VM VM VM

L3 L2

L3 L2

Access port (no VLAN tag)

OpenStack Networking – VLAN based •  The VLAN based model uses VLANs per tenant network (with Neutron) to provide

multi-tenancy, L2 isolation and support for overlapping IP address spaces

•  The VLANs can either be pre-configured manually on the physical switches, or a neutron vendor plugin can communicate with the physical switches to provision the VLAN

•  Examples of vendor plugins that are creating VLANs on Switches are the Arista and Cisco Nexus/UCS ML2 mechanism driver

•  L3 first-hop routing can be done either;

•  On the physical switches/routers, or

VM VM VM VM

VM VM VM VM

VM VM VM VM

L3 L2

L3 L2

VLAN trunk port (VLAN tags used)

VM VM VM VM

Neutron vendor plugin can create VLANs through vendor API

OpenStack Networking – VLAN based

VM VM VM VM

VM VM VM VM

VM VM VM VM

L3 L2

L3 L2

VLAN trunk port (VLAN tags used)

Logical routers are handling the first-hop gateway function on Neutron Network-Node

•  The VLAN based model uses VLANs per tenant network (with Neutron) to provide multi-tenancy, L2 isolation and support for overlapping IP address spaces

•  The VLANs can either be pre-configured manually on the physical switches, or a neutron vendor plugin can communicate with the physical switches to provision the VLAN

•  Examples of vendor plugins that are creating VLANs on Switches are the Arista and Cisco Nexus/UCS ML2 mechanism driver

•  L3 first-hop routing can be done either;

•  On the physical switches/routers, or •  As logical routers in

Neutron

Neutron vendor plugin can create VLANs through vendor API

L3 for tenant networks

VM VM VM VM

OpenStack Networking Models – ‘SDN Fabric’ based •  In this model multi-tenancy is achieved using different ‘edge’ and ‘fabric’ tags.

E.g. VLANs can be used to address the tenant between the hypervisor vSwitch and the Top-of-Rack switch, and some other tag is used inside of the vendors fabric to isolate the tenants

VM VM VM VM VM VM VM VM

Vendor Fabric uses some form of ‘Fabric Tag’ to address the tenant

VM VM VM VM VM VM VM VM VM VM VM VM

Hypervisor to Top of Rack Switch uses some form of ‘edge tag’ (e.g. VLAN, VXLAN header, etc.)

Central controller controls the vSwitches and physical Switches

Controller

•  Usually a single controller controls both the vSwitches and the physical switches

•  L3 first-hop routing and L2 bridging to physical usually done in the physical switch fabric

•  Single vendor design for physical and virtual networking

•  Examples; BigSwitch, NEC, Cisco ACI, Brocade, Avaya, … Neutron vendor plugin talks to controller through vendor API

Fabric Tag

Edge Tag Edge Tag

OpenStack Networking Models – Network Virtualization •  With network virtualization (aka overlay) model, multi-tenancy is achieved by overlaying

MAC-in-IP ‘tunnels’ onto the physical switching fabric (aka transport network)

•  An ID field is used in the encapsulation header (e.g. VXLAN, GRE, STT) to address the tenant network. A full L2 isolation and overlapping IP space support is achieved

•  Controller controls only the vSwitches and the Gateways

•  L3 first-hop routing and L2 bridging to physical done either by software or hardware gateways (or both)

•  Examples; VMware NSX, Midokura, Plumgrid, Contrail, Nuage, …

OpenStack DACH Day 2014 @ Linux Tag Berlin, 09.05

VM VM VM VM VM VM VM VM VM VM VM VM

VM VM VM VM VM VM VM VM

Physical network fabric uses L3 routing protocols (e.g. OSPF or BGP) to build a stable Layer 3 Fabric

SDN controller cluster controls the vSwitches in the Hypervisors

MAC-in-IP ‘Tunnel’ is used to address and isolate the tenants (e.g. using VXLAN)

L3 Gateway

L3 L2

L3 L2

L3 L3

L3 L2

Neutron plugin talks to controller through vendor API

Why I think the ‘Network virtualization’ (aka overlay) approach is the best model

•  It achieves multi-tenancy, L2 isolation and overlapping IP address support without the need to re-configure physical network devices

•  Logical Network for Instances (VMs) is location independent – It spans over L2/L3 boundaries, and therefore doesn’t force bad (flat) network design

•  Very big ID space for tenant addressing compared to the usual VLAN id space (max. 4094)

•  Network virtualization runs as a software construct on top of any physical network topology, vendor, etc.

•  Physical network and logical network can evolve independently from each other, each one can be procured, exchanged, upgraded and serviced independently

•  Large number of commercial and open source implementations are available today

•  Proven in production in some of the largest OpenStack deployments out there

Nova-Networking quick Overview nova-api

(OS,EC2,Admin) nova-console (vnc/vmrc)

nova-compute

Nova DB

nova-scheduler

nova-consoleauth

Hypervisor (KVM, Xen,

etc.)

Queue

nova-cert

Libvirt, XenAPI, etc.

nova-metadata

nova-network

nova-volume

Network-Providers (Linux-Bridge or OVS with

brcompat, dnsmasq, IPTables)

Volume-Provider (iSCSI, LVM, etc.)

Inspired by Ken Pepple

•  Nova-Networking was OpenStack’s first network implementation

•  Nova-network is still present today, and can be used instead of Neutron

•  No new features are added since Folsom, but bug-fixing is done frequently

•  Nova-network only knows 3 basic Network-Models;

–  Flat & Flat DHCP: direct bridging of Instance to external ethernet Interface with and without DHCP

–  VLAN based: Every tenant gets a VLAN, DHCP enabled

•  Watch this online meetup Session for more details: https://www.youtube.com/watch?v=ascEICz_WUY

Nova-Networking Multi-Host mode 1/2

nova-compute

hypervisor VM VM

Bridge 30 IP Stack

Compute Node + Networking

nova-compute

hypervisor VM VM

Br 30 IP Stack

Compute Node

nova-compute

hypervisor VM VM

IP Stack

Compute Node

External Network

(or VLAN)

Internal VLANs

WAN/Internet

dnsmasq

iptables/ routing

Bridge 40

VLAN30 VLAN40

Br 40

VLAN30 VLAN40

Br 30

Br 40

VLAN30 VLAN40

VLAN Trunk VLAN Trunk

dnsmasq

NAT & floating

-IPs

nova-netw.

•  In Nova-Networking the node that is holding the nova-networking role is; –  A single point of failure –  A choke point for both east-west and north-south traffic

(traffic staying in the DC between nodes and traffic leaving/entering the DC at the perimeter) –  Nova-Networking has a “multi-host mode” to address this

Nova-Networking Multi-Host mode 2/2

nova-compute

hypervisor VM VM

Bridge 30 IP Stack


External Network

(or VLAN)

Internal VLANs

WAN/Internet

dnsmasq

iptables/ routing

Bridge 40

VLAN30 VLAN40

VLAN Trunk VLAN Trunk

dnsmasq

NAT & floating

-IPs

nova-netw.

•  With nova-networking “Multi-Host” each compute node runs nova-networking, and provides routing, SNAT and floating-ip’s (DNAT) for its local Instances –  Pros; Inherently highly-available; scales out routing and NAT to all compute-nodes –  Cons; IP address sprawl: each compute-node needs one external IP for SNAT, and one internal IP

in each project Network

nova-compute

hypervisor VM VM

Bridge 30 IP Stack


dnsmasq

iptables/ routing

Bridge 40

VLAN30 VLAN40

dnsmasq

NAT & floating

-IPs

nova-netw. nova-compute

hypervisor VM VM

Bridge 30 IP Stack


dnsmasq

iptables/ routing

Bridge 40

VLAN30 VLAN40

dnsmasq

NAT & floating

-IPs

nova-netw.

External network

Nova-Networking vs. Neutron at a glance

•  Watch this online meetup Session for more details: https://www.youtube.com/watch?v=ascEICz_WUY

•  Neutron pros –  More network implementation options –  Dynamic network, virtual router, load

balancer, VPN creation under the tenants control instead of fixed per project allocation

–  Pluggable architecture allows vendors to integrate their network solution into OpenStack and innovate independently (e.g. using network virtualization, SDN concepts, etc.)

–  Well defined tenant accessible API for consuming network services

•  Nova-Networking pros –  Simple models with less moving parts –  “Compute centric” networking model;

easier to understand than the complex options and “networking speech” in Neutron

–  Code-Base is in “bug-fixing” mode since long time now; less friction

–  HA and scale-out trough “multi-host” option (starting to be addressed in Neutron by DVR and HA)

OpenStack Neutron – Plugin Concept refresher

Neutron Core API"

Neutron Service (Server)""

•  L2 network abstraction definition and management, IP address management

•  Device and service attachment framework •  Does NOT do any actual implementation of abstraction

"

Plugin API"

"Vendor/User Plugin"

•  Maps abstraction to implementation on the Network (Overlay e.g. NSX or physical Network) •  Makes all decisions about *how* a network is to be implemented •  Can provide additional features through API extensions. •  Extensions can either be generic (e.g. L3 Router / NAT), or Vendor Specific

"

Neutron API Extension"

Extension API implementation is optional

Core and service plugins •  Core plugin implement the “core” Neutron API functions

(l2 Networking, IPAM, …)

•  Service plugins implements additional network services (l3 routing, Load Balancing, Firewall, VPN)

•  Implementations might choose to implement relevant extensions in the Core plugin itself

Neutron Core API"Function"

Core "

L3 "

FW "

Core "

L3 "

FW "

Core "

L3 "

FW "

Plugin"Core Plugin

"

Core Plugin "

FW plugin

"

Core Plugin

"

FW plugin

"

L3 plugin

"

OpenStack Neutron – Plugin locations

!# cat /etc/neutron/neutron.conf | grep "core_plugin"!core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin!!# cat /etc/neutron/neutron.conf | grep "service_plugins”!service_plugins = neutron.services.l3_router.l3_router_plugin.L3RouterPlugin!!

!# ls /usr/share/pyshared/neutron/plugins/!bigswitch cisco embrane __init__.py metaplugin ml2 nec openvswitch ryu!brocade common hyperv linuxbridge midonet mlnx nicira plumgrid!!# ls /usr/share/pyshared/neutron/services/!firewall __init__.py l3_router loadbalancer metering provider_configuration.py service_base.py vpn""

OpenStack Neutron – Modular Plugin •  Before the modular plugin (ML2), every team or vendor had to implement a complete plugin

including IPAM, DB Access, etc.

•  The ML2 Plugin separates core functions like IPAM, virtual network id management, etc. from vendor/implementation specific functions, and therefore makes it easier for vendors not to reinvent to wheel with regards to ID Management, DB access …

•  Existing and future non-modular plugins are called “monolithic” plugins

•  ML2 calls the management of network types “type drivers”, and the implementation specific part “mechanism drivers”

Arista

Cisco Linux Bridge

OVS etc.

Mechanism

Drivers"

GRE

VLAN

VXLAN

etc. Type

Drivers"

Type Manager" Mechanism Manager "

ML2 Plugin & API Extensions"

OpenStack Neutron ML2 – locations

!# cat /etc/neutron/plugins/ml2/ml2_conf.ini | grep type_drivers! # the neutron.ml2.type_drivers namespace.! # Example: type_drivers = flat,vlan,gre,vxlan! type_drivers = gre!!# cat /etc/neutron/plugins/ml2/ml2_conf.ini | grep mechanism_drivers! # to be loaded from the neutron.ml2.mechanism_drivers namespace.! # Example: mechanism_drivers = arista! # Example: mechanism_drivers = cisco,logger! mechanism_drivers = openvswitch,linuxbridge!!

!# ls /usr/share/pyshared/neutron/plugins/ml2/drivers/!cisco l2pop mechanism_ncs.py mech_hyperv.py mech_openvswitch.py type_gre.py type_tunnel.py type_vxlan.py __init__.py mech_agent.py mech_arista mech_linuxbridge.py type_flat.py type_local.py type_vlan.py!"

OpenStack Neutron – Modular Plugin vs. Monolithic Plugins

•  A vendor is free to choose between the development of an monolithic plugin or an ML2 mechanism driver –  A vendor might want use its own integrated IPAM / DB access, or already has a stable and proven

code base for it –  Timing: Development of a monolithic plugin might have started long before ML2 emerged

•  Contrary to a common misunderstanding monolithic plugins are not deprecated, only the existing OVS-Plugin and Linux Bridge plugins have been deprecated in IceHouse in favor of the OVS / Linux Bridge mechanism drivers

•  ML2 re-uses the monolithic OVS and Linux Bridge code for its mechanism driver and agents (e.g L3 Agent, DHCP Agent, OVS Agent, etc.)

Juno – Distributed Virtual Router for OVS – 1/5 •  There is was equivalent of nova-network “multi-host” mode in Neutron before Juno •  In the OVS and Linux Bridge implementations, the L3 Agent node is a single point of failure.

•  Scaling out is done by deploying multiple network nodes, but even then east-west traffic needs to go through the L3 Agent Node, and can potentially be a choke point

•  Some vendor implementation already have distributed routing an HA today in their solutions

IP Stack

Neutron- Network-Node

nova-compute

hypervisor VM VM

IP Stack

Compute Node

nova-compute

hypervisor VM VM

Compute Node

External Network

(or VLAN)

WAN/Internet

iptables/ routing

Layer 3 Transport Network

dnsmasq NAT & floating

-IPs iptables/ routing

N.-L3-Agent N.-DHCP-Agent N.-OVS-Agent

ovsdb/ ovsvsd

Neutron-Server + OVS-Plugin

N.-OVS-Agent N.-OVS-Agent

ovsdb/ ovsvsd

ovsdb/ ovsvsd

Layer 3 Transport Net.

IP Stack

br-int br-int br-tun

br-int br-tun

br-tun

L2 in L3 Tunnel

dnsmasq

br-ex

Juno – Distributed Virtual Router for OVS – 2/5 •  Similar to “multi-host” mode in nova-network, each compute node now has its own routing and

NAT service (internal router namespaces - ‘IR’ )

•  In contrast to nova-network “multi-host” mode : –  SNAT will be done on a centralized network-node to avoid IP address sprawl on the external network

(introducing a single point of failure that needs to be addressed through virtual routers HA later) –  All IRs use a single logical internal IP in the tenant networks, but have separate MAC addresses

IP Stack

Neutron- Network-Node

nova-compute

hypervisor VM VM

Compute Node

External Network

(or VLAN)

WAN/Internet

iptables/ routing

Layer 3 Transport Network

dnsmasq SNAT -IPs iptables/

routing

N.-L3-Agent N.-DHCP-Agent N.-OVS-Agent

ovsdb/ ovsvsd

Neutron-Server + OVS-Plugin

N.-OVS-Agent

IP Stack

br-int br-int br-tun br-tun

L2 in L3 Tunnel

dnsmasq

br-ex

N.-L3-(DVR)-Agent

iptables/ routing

NAT for floating

-IPs

iptables/ routing

br-ex

ovsdb/ ovsvsd

nova-compute

hypervisor VM VM

Compute Node

N.-OVS-Agent

IP Stack

br-int br-tun

iptables/ routing

NAT for floating

-IPs

iptables/ routing

br-ex

ovsdb/ ovsvsd

Layer 3 Transport Net.

External Network

(or VLAN)

External Network

(or VLAN)

N.-L3-(DVR)-Agent

br-int

br-int

Juno – Distributed Virtual Router for OVS – 3/5 •  For east-west traffic which is routed within a tenants distributed virtual router,

traffic is send directly between compute-nodes on the transport network (Using the overlay technology)

•  Traffic can also stay within a compute-node, if the source and destination are on the same compute node

•  For more details see the DRV blueprint: https://blueprints.launchpad.net/neutron/+spec/neutron-ovs-dvr

east-west

north-south

Com

pute

Nod

e

VM

VM

VM

VM

IR2

IR1

WAN/Internet

Com

pute

Nod

e

External Network

Transport Network (e.g. used for tunnels)

Net

wor

k N

ode

IR2

IR1 VM

VM

VM

VM

br-tun br-tun br-tun

br-ex br-ex br-ex

br-int

R2 / SNAT

R1 / SNAT

br-in

t

Juno – Distributed Virtual Router for OVS – 4/5 •  For SNAT from the tenant instances to the internet/WAN (north/south) traffic is

routed through a centralized network-node

•  This avoids IP address sprawl on the external network •  For more details see the DRV blueprint:

https://blueprints.launchpad.net/neutron/+spec/neutron-ovs-dvr

east-west

north-south

Com

pute

Nod

e

VM

VM

VM

VM

IR2

IR1

WAN/Internet

Com

pute

Nod

e

External Network


Net

wor

k N

ode

R2 / SNAT

R1 / SNAT

IR2

IR1 VM

VM

VM

VM

SNAT Router

-IP


br-ex br-ex br-ex

br-int

br-int

br-int

Juno – Distributed Virtual Router for OVS – 5/5 •  For floating-ip’s to and from the tenant instances to the internet/WAN (north/

south) traffic is routed and nat’ed directly at the compute nodes (IR Namespace)

•  For more details see the DRV blueprint: https://blueprints.launchpad.net/neutron/+spec/neutron-ovs-dvr

east-west

north-south

Com

pute

Nod

e

VM

VM

VM

VM

IR2

IR1

WAN/Internet

Com

pute

Nod

e

External Network


Net

wor

k N

ode

R2 / SNAT

R1 / SNAT

IR2

IR1 VM

VM

VM

VM

floating -IP


br-ex br-ex br-ex

br-int

br-int

Juno – Current caveats for Distributed Virtual Router

•  Currently there is no support for HA in the centralized SNAT Node (north/south). Although there is L3 Agent HA in Juno today, you need to choose between DVR mode or L3 HA today. The plan is to address this in Kilo, or even later, as the Neutron team has other technical debt to work on

•  No IPv6 Support

•  DVR is only supported with OVS Plugin with VXLAN based overlays, no support for VLAN modes and/or for Linux Bridge Plugin

•  No support for VPNaaS

•  Longer term plans –  Distributed SNAT –  Distributed DHCP (nova-network has this today) –  Full migration support from virtual routers to DVR

Juno – HA for Virtual Routers

•  Juno added native HA support using ‘keepalived’ for the centralized L3 agent nodes •  If configured for HA, one active and one standby router will be deployed on two different

neutron L3 GW network nodes. Both will share Virtual IPs internally •  For more details see the HA for virtual routers blueprint:

https://github.com/openstack/neutron-specs/blob/master/specs/juno/l3-high-availability.rst

+----+ +----+! | | | |!+-------+ QG +------+ +-------+ QG +------+!| | | | | | | |!| +-+--+ | | +-+--+ |!| VIPs| | | |VIPs |!| | +--+-+ +--+-+ | |!| + | | | | + |!| KEEPALIVED+---+ HA +------+ HA +----+KEEPALIVED |!| + | | | | + |!| | +--+-+ +--+-+ | |!| VIPs| | | |VIPs |!| +-+--+ | | +-+--+ |!| | | | | | | |!+-------+ QR +------+ +-------+ QR +------+! | | | |! +----+ +----+!

Juno – Current caveats for L3 Agent HA

•  Currently there is no state synch for NAT tables and FWaaS states, planed to be address in Kilo or later using Conntrackd

•  No support for HA when using the DVR functionality (also goes with the first bullet) •  No logging for state transitions, no CLI to see where the active router is and no CLI to move it

between nodes

•  Currently no automatic migration of existing routers to HA routers •  Max. 255 router pairs per HA network, and therefore per tenant

Juno – IPv6 support

•  IPv6 was dysfunctional at multiple implementation points in Neutron before Jun0 –  No support for Stateless Auto Configuration (SLAAC) in OpenStack security model / IPAM, so

even when one uses an external IPv6 router, security groups and port security will prevent the Instance from working correctly

–  Dnsmasq support for DHCPv6 was problematic and “broken” –  No IPv6 Routing support on L3 Agent, Metadata, etc.

•  A new IPv6 Neutron Subteam was founded to address the multiple IPv6 requirements •  Expected critical IPv6 Features in Juno Timeframe

–  Provider Networking - upstream SLAAC Support –  Support DHCPv6 stateless and stateful mode in Dnsmasq –  Support Router Advertisement Daemon (radvd) for IPv6

•  See more details here: https://wiki.openstack.org/wiki/Neutron/IPv6

Juno – More Information

•  A big number of new vendor plugins, enhancements to existing plugins and mechanism drivers, service plugins etc. are being developed for the Juno timeframe right now

•  See here for a list of Juno Specs (linking to the Blueprints): https://github.com/openstack/neutron-specs/tree/master/specs/juno

•  See here for a list of Blueprints: https://blueprints.launchpad.net/neutron/juno

Questions?

Open stack networking_101_update_2014-os-meetups

Technology

Transcript of Open stack networking_101_update_2014-os-meetups