Post on 16-Apr-2017
Betting on ContainerNetworking
Lee CalcoteJuly 27th, 2016
OverlayUnderlay
http://calcotestudios.com/over-under
http://calcotestudios.com/oubcn
Lee Calcoteclouds, containers, infrastructure, applications
and their management
linkedin.com/in/leecalcote
@lcalcote
blog.gingergeek.com
lee@calcotestudios.com
Show of Hands
@lcalcote
Preset Expectations
Experience &Management
Reliability &Performance
same demands and measurements
developer-friendly and application-driven
simple to use and deploy for developers and operators
better or at least on par with their existing virtualized datacenter networking
Very interesting
but no need to actually know these
@lcalcote
(CNM)
Container Network Model
...is a specification proposed by Docker,
adopted by projects such as
Plugins built by projects such as ,
, and
libnetwork
Weave
Project Calico Kuryr
(CNI)
Container Network Interface
...is a specification proposed byCoreOS and adopted by projects suchas , , ,
, and Plugins created by projects such as
, , and
rkt Kurma Kubernetes CloudFoundry Apache Mesos
Weave Project Calico ContivNetworking
@lcalcote
Container Networking Specifications
Container Network ModelSpecification
@lcalcote
Remote DriversLocal Drivers
Container Network ModelTopology
@lcalcote
Network Sandbox
Endpoint
Backend Network
DockerContainer
Network Sandbox
Endpoint
DockerContainer
Network Sandbox
Endpoint
DockerContainer
Endpoint
Frontend Network
@lcalcote
Container Network InterfaceFlow
1. Container runtime needs to:
1. allocate a network namespace to the container and assign a container ID
2. pass along a number of parameters (CNI config) to network driver.
2. Network driver attaches container to a network and thenreports the assigned IP address back to the container runtime(via JSON schema)
@lcalcote
CNI Network(JSON)
{ "name": "mynet", "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "ipam": { "type": "host-local", "subnet": "10.22.0.0/16", "routes": [ { "dst": "0.0.0.0/0" } ] }
CNI and CNMSimilar in that each...
...are driver-based, and therefore
democratize the selection of which type of container networking
...allow multiple network drivers to be active and usedconcurrently
each provide a one-to-one mapping of network to that network’s driver
...allow containers to join one or more networks.
...allow the container runtime to launch the network in itsown namespace
segregate the application/business logic of connecting the container to
the network to the network driver.
@lcalcote
CNI and CNMDifferent in that...
CNI supports any container runtime
CNM only support Docker runtime
CNI is simpler, has adoption beyond its creator CNM acts as a broker for conflict resolution
CNI is still considering its approach to arbitration
@lcalcote
Types of Container Networking
NoneLinks and AmbassadorsContainer-mappedBridgeHostOverlay
Underlay
MACvlanIPvlanDirect Routing
Point-to-PointFan Networking
@lcalcote
None
@lcalcote
container receives a network stack, butlacks an external network interface.
it does, however, receive a loopback
interface.
Linksfacilitate single host connectivity
"discovery" via /etc/hosts or env vars
@lcalcote
Ambassadorsfacilitate multi-host connectivity
uses a tcp port forwarder (socat)
Web Host
MySQL
AmbassadorPHP
DB Host
PHP
AmbassadorMySQL
link link
Container-Mappedone container reuses (maps to) the networking
namespace of another container.
@lcalcote
may only be invoked when running a dockercontainer (cannot be defined in Dockerfile):
--net=container=some_container_name_or_id
BridgeAh, yes, docker0
default networking for Dockeruses a host-internal networkleverages iptables for network address translation(NAT) and port-mapping
@lcalcote
Hostcontainer created shares its network namespace with the host
default Mesos networking modebetter performance easy to understand and troubleshootsuffers port conflictssecure?
@lcalcote
Overlayuse networking tunnels to delivery communication
across hosts
Most useful in hybrid cloud scenariosor when shadow IT is needed
Many tunneling technologies existVXLAN being the most commonly used
Requires distributed key-value store
@lcalcote
K/V Store for OverlayNetworking
Docker - requires K/V store (built-in as experimental as of1.12)
WeaveMesh - does not require K/V store
WeaveNet - limited to single network; requires K/V store
Flannel - requires K/V store
Plumgrid - requires K/V store; built-in and not pluggable
Midokura - requires K/V store; built-in and not pluggable
Calico - requires K/V store
@lcalcote
Underlaysexpose host interfaces (i.e. the physical network interface at
eth0) directly to containers running on the host
MACvlanIPvlanDirect Routing
@lcalcote
not necessarily public cloud friendly
Point-to-Point
Default rkt networking modeUses NAT (IPMASQ) by defaultCreates a virtual ethernet pair
placing one on the host and the other into the container pod
leverages iptables to provide port-forwardingfor inbound traffic to the podinternal communication between othercontainers in the pod over the loopbackinterface
@lcalcote
Internet
MACvlanallows creation of multiple virtual network interfaces behindthe host’s single physical interface
Each virtual interface has unique MAC and IP addressesassigned
with restriction: the IP address needs to be in the same broadcast
domain as the physical interface
eliminates the need for the Linux bridge, NAT and port-mapping
allowing you to connect directly to physical interface
@lcalcote
IPvlanallows creation of multiple virtual network interfaces behind the host’s single
physical interface
Each virtual interface has unique IP addresses assigned
Same MAC address used for all containers
L2-mode containers must be on same network as host (similar to MACvlan)
L3-mode containers must be on different network than host
Network advertisement and redistribution into the network still needs to be done.
@lcalcote
MACvlan and IPvlanWhile multiple modes of networking are supported on a given host, MACvlan
and IPvlan can’t be used on the same physical interface concurrently.
ARP and broadcast traffic, the L2 modes of these underlay drivers operate
just as a server connected to a switch does by flooding and learning using
802.1d packets
IPvlan L3-mode - No multicast or broadcast traffic is allowed in.
In short, if you’re used to running trunks down to hosts, L2 mode is for you.
If scale is a primary concern, L3 has the potential for massive scale.
@lcalcote
Direct RoutingBenefits of pushing past L2 to L3
resonates with network engineers
leverage existing network infrastructure
use routing protocols for connectivity; easier to interoperate with existingdata center across VMs and bare metal servers
Better scaling
More granular control over filtering and isolating network traffic
Easier traffic engineering for quality of service
Easier to diagnose network issues
@lcalcote
Fan Networkinga way of gaining access to many more IP addresses, expanding from one assigned IP
address to 250 more IP addresses “address expansion” - multiplies the number of available IP addresses on the
host, providing an extra 253 usable addresses for each host IP
Fan addresses are assigned as subnets on a virtual bridge on the host,
IP addresses are mathematically mapped between networks
uses IP-in-IP tunneling; high performance
particularly useful when running containers in a public cloud
where a single IP address is assigned to a host and spinning up additional networks is prohibitive or
running another load-balancer instance is costly
@lcalcote
Fan Networking
@lcalcote
Network Capabilities andServices
IPAM, multicast, broadcast, IPv6, load-balancing, service discovery, policy, quality
of service, advanced filtering and performance are all additional considerations to
account for when selecting networking that fits your needs.
@lcalcote
IPv6 and IPAMIPv6
lack of support for IPv6 in the top public clouds
reinforces the need for other networking types (overlays and fan networking)
some tier 2 public cloud providers offer support for IPv6
IPAM
most container runtime engines default to host-local for assigning addresses
to containers as they are connected to networks.
Host-local IPAM involves defining a fixed block of IP addresses to be selected.
DCHP is universally supported across the container networking projects.
CNM and CNI both have IPAM built-in and plugin frameworks for integration
with IPAM systems@lcalcote
Text
@lcalcote
Docker 1.12 (Load-balancing)
Lee Calcoteclouds, containers, infrastructure, applications
and their management
linkedin.com/in/leecalcote
@lcalcote
blog.gingergeek.com
lee@calcotestudios.com
Thank you!Questions?