Docker IP Routing PLACE IMAGE OVER THIS SPACEfiles.meetup.com/10524692/Docker Routed...

Medallia © Copyright 2015. 1

PLACE IMAGE OVER THIS SPACE

Docker IP RoutingHaving your first-hop load-balancer on Docker


Who are you?

Medallia:“Software to improve the customer experience”“Aggregating 1B documents in 1s or less”

This talk: Infrastructure to run the crunching application

[email protected]


• Docker for everything!• Docker for applications!• Docker for load-balancers!• Docker for zookeeper! And DNS!

Problem to solveWant a reliable, flexible data-center


Problem to Solve TodayMoving non-movable services

DataCenter Firewall

Host: 10.1.2.3:80

Host: 10.1.2.5:80

172.17.0.3:80 nginxHost: 10.1.2.4:2181

172.17.1.0:2181 zookeeper

172.17.1.2:80 application


• Buy the cheapest possible servers○ Optimize for performance○ Sacrifice redundancy○ Service Contract: “Unrack and ship by mail”

• Solve the redundancy at a higher level○ Run multiple instances of everything○ Rapidly restore status quo

Design for FailureIt will fail sooner or later; choose sooner


• No special snowflakes• Commodity Components & Supported Open Standards• Fully automated provisioning and reinstall• Cheap• Scalable• “Simple”• Every component must be able to run anywhere

Design Principles


Going to talk about Network today.

(Storage next time)

Two ProblemsStorage and Network


• VLAN○ Can’t do leaf/spine; need monstrous “core switches”

• VXLAN○ Network Island: How do you exit it?○ Vendor interoperability for forwarding DB (VXLAN-to-IP)

Time to leave layer 2 behind?

Things we triedThat didn’t work for us


Docker Bridged Networking Model

host1

container 1

eth0

10.1.2.3/24

172.16.1.1172.16.1.2

docker0

veth0 veth1


Default (Bridged) Strategy

● Creates a pair of veth.● Moves one to the container

namespace.● Renames the container veth to

eth0● Attaches the host veth to the

docker0 bridge● Configure port forwarding in

iptables

Routed Strategy

● Creates a pair of veth.● Moves one to the container

namespace.● Renames the container veth to

eth0.● Add route to 0.0.0.0/0 via

eth0 in container.● Add route to container IP via

veth0 in the host.

Docker* new strategy


OSPF Area

host1% ip route10.4.5.6 dev veth0

...

Docker* Routed Networking Model

eth0

10.1.2.3/24

veth0

container-A

% ip routedefault eth0

10.4.5.6/32

eth0

host2

host3

hostN


Route to 10.1.2.3/32Infrastructure

Spine

Leaf

Server

10.1.2.3/32

10.1.2.3/32


Servers, Network, OSPFInfrastructure

Spine

Leaf

Server


OSPF: 1998

• Open Shortest Path First○ Propagated Link State Database○ Supported by every vendor

• OSPF is computationally expensive○ On a 1998-style embedded controller: Yes○ On a 2015-style Intel Atom 64-bit: No

• Everything is point-to-point L3 links• Switches and Servers run OSPF (Quagga)• Cumulus! OSPF unnumbered

Old and boring is the new sexy


Running a Container(technically, create and start a container)

% docker run -it --name=foo --net=routed --ip-address=10.2.3.4/32 ubuntu /bin/bash

(Will likely change to use labels)


demo!


Pros and ConsAwesomeness SucksiesIP MobilityQuick failoverNo special snowflakesEverything in docker

Future: CRIU?

T2 routing limit: 128K entries


Leaf /etc/network/interfaces Server /etc/network/interfacesauto loiface lo inet loopback address 10.225.10.245/32%for v in range(1,17):auto swp${v}iface swp${v} mtu 9000 address 10.225.10.${v*8+1}/30%endfor%for v in range(17,33):auto swp${v}iface swp${v} mtu 9000 address 10.225.10.245/32%endfor

auto loiface lo inet loopback

auto data0iface data0 inet static mtu 9000 address 10.225.10.10 netmask 255.255.255.252 gateway 10.225.10.9

How difficult is the network config?

To Servers

To Spines

To Leaf


Leaf/Spine Switch ospfd.conf Server ospfd.confrouter ospf ospf router-id 10.225.10.245 network 10.224.0.0/12 area 0.0.0.0!interface swp1 ip ospf network point-to-point!interface swp2 ip ospf network point-to-point!….

! Bootstrap Configrouter ospf ospf router-id 10.225.10.10 redistribute kernel passive-interface default no passive-interface data0 network 10.224.0.0/12 area 0.0.0.0!log syslog!interface data0 ip ospf network point-to-point!

How difficult is the network config?


“Good enough”

• 24-39 Gbit/s (core affinity)• 13us ICMP ping

Performance


Local Development With Style

IP Mobility on Local Laptop

Allow for easy and rapid development

Boot2Docker


Front-End (On 10.10.2.1)

What day is today as 2 servicesBack-End (On 10.10.2.2)while true; do

date | nc -l 9999done

while true; doecho Today is $(nc 10.10.2.2 9999) | nc -l 8080

done


Boot2Docker on OSX

My MacBookBoot2Docker VM

10.10.0.0/16

10.10.0.0/16

10.10.2.1/32 frontend on 8080

10.10.2.2/32 backed (date) on 9999

lo0: 10.10.2.2/32 backend

My Shell


demo!


Next Steps for us

• Share with the world!○ Everything you’ve seen today is (or will be) open source

• Storage with CEPH○ Already works in medallia-container (think systemd-nspawn)○ Porting to docker

• SSH Hot Redirect○ SSH to container => SSH to host with automatic “docker exec”

• Docker Labels in Aurora/Mesos

In the next sprint...


Checkout and have fun

www.github.com/medallia/dockerwww.github.com/medallia/boot2docker-iso

[email protected]


Questions?

Docker IP Routing PLACE IMAGE OVER THIS SPACEfiles.meetup.com/10524692/Docker Routed...

Documents

Transcript of Docker IP Routing PLACE IMAGE OVER THIS SPACEfiles.meetup.com/10524692/Docker Routed...