Neutron L3 Agent HAOr: How I Learned to Stop Worrying and Love the API

Kevin Bringard // OpenStack Juno Summit // May 2014

• There is no “one right way” • The goal is to move L3 resources to a new L2

resource as quickly and seamlessly as possible • This is a really difficult, but important, problem to

solve

Layer 3Internet Happens

L3 agent L3 agent L3 agent

router1 router2router3router4router5

router6

VM1 VM3VM2 VM4 VM5 VM7VM6

Core Router

L3 agent L3 agent

router1 router2router3router4router5

router6

VM1 VM3VM2 VM4 VM5 VM7VM6

Core Router

Layer 2The ARPing is the hardest part

• One L3 resource may only be tied to one L2 resource at a time

• Many technologies exist to sort of work around this • HSRP • VRRP • CARP

• Work is being done to implement VRRP like functionality into Juno • https://blueprints.launchpad.net/neutron/+spec/l3-

high-availability • Nothing is currently integrated into OpenStack

Pacemakerhttp://docs.openstack.org/high-availability-guide/content/_highly_available_neutron_l3_agent.html

• False positives — caused more downtime than actual outages

• Split brain possibilities • Assumes control of L3 agent start/stop functions • Limited Horizontal Scale

• More difficult to run multiple Active L3 agents • Failover requires entire services starts/stops

• Active/Passive Model Requires More Hardware • Works on a “per agent” level • Akin to RAID1

L3 agent L3 agent L3 agent

router1 router2router3router4router5

router6

VM1 VM3VM2 VM4 VM5 VM7VM6

Core Router

L3 agent L3 agent

router1 router2router3router4router5

router6

VM1 VM3VM2 VM4 VM5 VM7VM6

Core Router

L3 agent L3 agent L3 agent

router1 router2router3router4router5

router6

VM1 VM3VM2 VM4 VM5 VM7VM6

Core Router

Neutron HA Toolhttps://raw.githubusercontent.com/stackforge/cookbook-openstack-network/master/files/default/neutron-ha-tool.py

• API Driven • Uses native API calls to perform all functions • Can be run externally from infrastructure or cross

site • Supports any operations the neutron client

libraries supports • Easily Extendable

• Written in python • Leverages standard OpenStack libraries

• Works on a “per resource” level

L3 agent L3 agent L3 agent

router1 router2router3router4router5

router6

VM1 VM3VM2 VM4 VM5 VM7VM6

Core Router

L3 agent L3 agent

router1 router2router3router4router5

router6

VM1 VM3VM2 VM4 VM5 VM7VM6

Core Router

L3 agent L3 agent

router1 router2

router3router4

router5

router6

VM1 VM3VM2 VM4 VM5 VM7VM6

Core Router

• Only routers/IPs on the affected L3 agent are impacted

• Recovery time depends on the number of routers which need to be migrated and the number of IPs on each router

• Migration happens quickly, but every IP on the routers must re-ARP to the upstream switch

• Meta-data proxies migrate with the routers

OK, so what’s the catch?

• Not seamless • The ARP processes happen in parallel, but generally

take 60-90 seconds for all IPs to complete • Various *aaS offerings further complicate things

• Currently only accounts for “l3-agent” controlled services

• No coordination between HA tools • How do you HA the HA?

• Currently not daemonized, runs from cron • Add 60 seconds to total recovery time • Jitter protection adds additional total recovery time

• No mechanism by which to ensure resources actually come up/work

What about DHCP?

• Multiple DHCP agents may be run Active/Active • DHCP agents per subnet may be specified in your

agent config file • Each agent requires an IP in the tenant’s subnet • DHCP is multi-cast

• All agents have the same lease file • The first one to reply binds to the VM

• Any DHCP agent may reply to a DNS request and resolve all known leases

• By default, each DHCP agent hands out a list of every agent as available resolvers

• HA tool has an option to replicate DHCP to all agents

• VRRP Like functionality • Specify number of Active L3 agents per subnet • Leverage conntrackd/keepalived • Point of diminishing returns for HA tool? • The beauty of open source:

• There is no “one right way” • Think outside the box • Do cool things

Moving Forward

Questions?