in Kubernetes

Alban Crequy

Using Traffic Controlto Test Apps

KubeCon EU 2016 - Londonhttps://goo.gl/Zh2CMQ

Alban Crequy

∘ Worked on rkt the last 14 months∘ Currently tech lead on rkt∘ In 2014, worked on traffic control for multimedia

applications in cars (tcmmd)

https://github.com/alban

∘ What is traffic control and how does it work on Linux∘ Using TC in containers for tests

∘ Demo∘ In Kubernetes

∘ Demo with pings∘ Demo with guestbook

∘ Integration in a testing framework∘ Demo with guestbook

Plan

What is traffic control?

How does it work on Linux?

Traffic control, why?

web server client

client

client

THEINTERNET

∘ fair distribution of bandwidth

∘ reserve bandwidth to specific applications

∘ avoid bufferbloat

∘ Network scheduling algorithm∘ which packet to emit next?∘ when?

∘ Configurable at run-time:∘ /sbin/tc∘ Netlink

∘ Default on new network interfaces: sysctl net.core.default_qdisc

Queuing disciplines (qdisc)

eth0 THE INTERNETqdisc

∘ First In, First Out∘ But with 3 bands, based on IP header’s ToS field (type of service)

Linux’ default qdisc: pfifo_fast

eth0 THE INTERNETFIFO 1

FIFO 2

FIFO 0

Stochastic Fairness Queueing (sfq)

eth0

THE INTERNET

FIFO n

FIFO 1

FIFO 0

...

round robin

∘ drop packets to avoid buffer bloat∘ similar to Random Early Detection (red) but based on delays rather than

the size of the buffer∘ set as default by systemd since 2014

Fair Queuing Controlled Delay (fq_codel)

eth0 THE INTERNETX

Traffic control for testing?

Network emulator (netem)

eth0 THE INTERNETnetem

bandwidth

latency packet loss

corrupt...

Testing with containers

container 1 container 2

eth0eth0

Testing framework

configure “netem” qdiscs:bandwidth, latency, packet drop...

ingress / egress

server THE INTERNET

egress

ingress

ingress / egress

eth0 THE INTERNET

egress

ingress

ifb0

Testing a media server

rkt pod host

veth1eth0

RTP server

egressqdisc

media player

Demo

Try it yourself: https://github.com/kinvolk/demo

How it worked

rkt pod host

veth1eth0

RTP server

egressqdisc

media player

In Kubernetes

Testing with traffic control in Kubernetes

Kubernetes minion 1

pod

pod

Kubernetes minion 2

pod

podTesting framework

∘ configure network simulator

∘ play scenarios

Testing with traffic control in Kubernetes

Kubernetes minion 1

pod

pod

Kubernetes minion 2

pod

pod

tcd tcdgRPC or D-Bus methods:∘ Install()∘ ConfigureEgress()

https://github.com/kinvolk/tcd

Testing Weave Scope

Kubernetes minion 1

tcdScopeProbe

pod pod

pod pod

Kubernetes minion 2

tcdScopeProbe

pod pod

pod pod

ScopeApp

Demo

Try it yourself: https://github.com/kinvolk/demo

Demo

Try it yourself: https://github.com/kinvolk/demo

Testing framework for web apps

Selenium

Demo

Testing more complex scenarios

(my wishlist)

How to define classes of traffic

eth0

netem

interface

latency=100ms, drop=2%

http/80ip=10.0.4.*

http/80ip=10.0.5.*

other,dns/53, ...

u32: filter on contenteth0

HTB

HTB

HTBHTB HTB

netemnetem netem

interface

root qdisc (type = HTB)

root class (type = HTB)

leaf qdiscs (type = netem)

leaf classes (type = HTB)

filters (type=u32)dport=80

dport=53ip=10.*

latency=10ms drop=2%

Using filters in Kubernetes

Kubernetes minion 1

pod

pod

Kubernetes minion 2

pod

podTesting framework

drop100%

latency100ms

latency100ms

configuring tc filterbased on IPs(type=u32)

Testing Raft Consensus Algorithm

etcd

etcd

etcd

etcd

latency1ms

latency80ms

latency5000ms

etcd parameters:

∘ heartbeat interval: 100ms∘ election timeout: 1000ms

∘ 1 network namespace per pod∘ rktnetes: apps started as

systemd units

∘ How to filter by app?systemd.resource-control(5):NetClass=auto∘ added in v227, 2015-10-07∘ removed in v229 :(

Filtering by app

Kubernetes minion 1

pod

app app

pod

app app

cgroup “net_cls”: filter by app

∘ Classifying based on cgroups with “net_cls”∘ Previously exposed by systemd∘ Then, tc filter “cgroup”∘ But not available in cgroup unified hierarchy, to ensure delegation∘ netfilter/iptables being replaced by nftables∘ New xt_cgroup just added to match on cgroup full path, then could

mark it and use net_cls

Filtering with cBPF/eBPF

eth0

BPF

netemnetem

kernel

userspace

BPF_JMP...BPF_LD...BPF_RET...

if (skb->protocol…) return TC_H_MAKE(TC_H_ROOT, mark); compilation

clang... -march=bpf

uploadin the kernel:

- bpf()- Netlink

x86_64 codeJIT compilation

eBPF maps

eth0

BPF

netemnetem

kernel

userspace

x86_64 code

eBPF map

Testing framework

∘ Build statistics∘ Make them available to

the testing framework

The EndTry the demos yourself: https://github.com/kinvolk/demo

Source: https://github.com/kinvolk/tcd

The slides: https://goo.gl/Zh2CMQ

2 things before questions

We’re Hiringhttps://kinvolk.io/careers/

in Berlin

coreos.com/fest - @coreosfestMay 9 & 10, 2016 - Berlin, Germany

Questions?