Post on 19-Feb-2017
+
A Cloud Native SDN for Kubernetes
Juergen Brendel, Stas KraevKubecon, London, March 2016
romana.io A cloud native SDN for Kubernetes @romanaproject
Agenda
● “Cloud native”, why does it matter?● A better network for cloud native architectures● New things in Kubernetes● Demos
romana.io A cloud native SDN for Kubernetes @romanaproject
About us
● Team background:– Data center networks
– Low-level traffic management
● Created L2 overlay network startup– Bought by Cisco
● OpenStack networking● There's got to be a better way
– Time is right
What is 'cloud native'?
romana.io A cloud native SDN for Kubernetes @romanaproject
The past: Enterprise networking
● Full control● Applications need L2 and L3
– May need hard-wired IP addresses
– Broadcasts
● Servers are pets, not cattle: “Careful!”– VM migration
● Complex!– Complexity in the applications
– Because apps may do anything, network needs to support everything!
romana.io A cloud native SDN for Kubernetes @romanaproject
Cloud native applications
● Automate all the things!– Infrastructure as code
– Cattle, not pets: “Meh... just kill it.”
– Workloads come and go quickly
– Build for resiliance
● IP is all you need– No hardcoded IP addresses, discovery
– No special network requirements
– Basic IP connectivity
● Restrictions– Accept them and get clarity and simplicity in return
The problem
romana.io A cloud native SDN for Kubernetes @romanaproject
We have a mismatch
● Building cloud native applications…● … on top of enterprise networking
– SDN controllers use overlay L2 domains
– VLAN, VXLAN, OVS, etc.
● Complexity and brittleness– Lose benefits of simplicity
– Lose performance (encap, blinded hardware)
– Difficult to maintain and trouble shoot
romana.io A cloud native SDN for Kubernetes @romanaproject
The price you pay: Complexity
VXLAN Decap
VXLAN Decap
VXLAN Encap
VXLAN Encap
2 Top of Rack Round Trips
East/West Traffic
Per Instance Security
romana.io A cloud native SDN for Kubernetes @romanaproject
The price you pay: Performance
Router
Endpoint A Endpoint B
Router
L2 overlay A
L2 overlay B
VRouter
romana.io A cloud native SDN for Kubernetes @romanaproject
Why do we do this to ourselves?
● We don't need any L2 features● Except traffic segmentation
– Multi tenancy
– Tiers and policies
The solution
romana.io A cloud native SDN for Kubernetes @romanaproject
Cloud native SDNs
● Use native L3 capabilities● No overlays● De-emphasize IP address ranges● Still provides segmentation, multi tenancy● Simple, clear and scalable network setup
romana.io A cloud native SDN for Kubernetes @romanaproject
A truly cloud native SDN: Romana
● Project Romana● Open source● Apache 2.0 license● Mostly written in Go● Kubernetes and OpenStack
romana.io A cloud native SDN for Kubernetes @romanaproject
A truly cloud native SDN: Romana
● Use only IP routing– No overlays
– All workload addresses are 'real'
– Simplicity!
● Use smart addressing– Encode tenant or segment in IP address
– Assign “virtual” addresses with host prefixes
– Massive (!) collapse of route table
● Routes are static– No route updates, no broadcasts for new endpoint
romana.io A cloud native SDN for Kubernetes @romanaproject
Routing and route aggregation
Host A
eth0:192.168.8.11
romana-gw:10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Host B
eth0:192.168.8.22
romana-gw:10.1.0.1/16
10.1.3.52
10.1.9.2
Host C
eth0:192.168.8.33
romana-gw:10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6
Routes:10.1/16 → 192.168.8.2210.2/16 → 192.168.8.33
Routes:10.0/16 → 192.168.8.1110.2/16 → 192.168.8.33
Routes:10.0/16 → 192.168.8.1110.1/16 → 192.168.8.22
romana.io A cloud native SDN for Kubernetes @romanaproject
Architecture
Host A Host B Host C
Agent Agent Agent
Tenant
Topology
IPAM
Root
Kubernetes
romana.io A cloud native SDN for Kubernetes @romanaproject
Architecture
Host A Host B Host C
Agent Agent Agent
Tenant
Topology
IPAM
Root
OpenStack
Romana / Kubernetes integration
romana.io A cloud native SDN for Kubernetes @romanaproject
Integration points
● CNI (Container Network Interface)– Developed last year by CoreOS
– Supported by Kubernetes since version 1.1
● Third party resources– Develop Kubernetes extensions via external
processes
● Network Policies– Still under development by networking SIG
– Different proposals under discussion
romana.io A cloud native SDN for Kubernetes @romanaproject
CNI_COMMAND (ADD | DEL)CNI_CONTAINERIDCNI_NETNSCNI_IFNAMECNI_ARGS...
CNI: Interface creation workflow
Host A
eth0:192.168.8.11
RomanaCNI plugin
Kubelet Create interface
romana.io A cloud native SDN for Kubernetes @romanaproject
CNI: Interface creation workflow
Host A
eth0:192.168.8.11
RomanaCNI plugin
Kubelet
RomanaIPAM
RomanaTenant
RomanaTopology
HostTenant
Segment
romana.io A cloud native SDN for Kubernetes @romanaproject
CNI: Interface creation workflow
Host A
eth0:192.168.8.11
RomanaCNI plugin
Kubelet
RomanaAgent
10.0.0.5
connectivity
policies
RomanaIPAM
RomanaTenant
RomanaTopology
IP address
romana.io A cloud native SDN for Kubernetes @romanaproject
Third party resources
● Tell Kubernetes about your new resource
$ kubectl create f thirdpartyresourcedefinition.yml
● Start listening for events on new URLs
/apis/romana.io/demo/v1/namespaces/default/networkpolicys/
metadata: name: networkpolicy.romana.ioapiVersion: extensions/v1beta1kind: ThirdPartyResourcedescription: "Network policy"versions: name: demo/v1
romana.io A cloud native SDN for Kubernetes @romanaproject
Kubernetes network polices
● Recognized need for policies– Grant / deny access, isolate tiers and tenants
– Basically: ACLs
– Different proposals exist
– Implementations use Kubernetes 3rd party resources
● Namespaces– Use namespace as 'tenant'
– Add 'isolation' flag to namespace
romana.io A cloud native SDN for Kubernetes @romanaproject
Example network policy
POST /apis/romana.io/demo/v1/namespaces/tenanta/networkpolicys/
{ "kind": "NetworkPolicy", "metadata": { "name": "pol1" }, "spec": { "allowIncoming": { "from": [ { "pods": { "segment": "frontend" } } ], "toPorts": [ { "port": 80, "protocol": "TCP" } ] }, "podSelector": { "segment": "backend" } }}
Gets applied to
namespace
“segments”: Natural fit
for Romana
romana.io A cloud native SDN for Kubernetes @romanaproject
Network policy workflow
Kubernetes master
Kubernetes API 3rd party resource type definitionkubectl
romana.io A cloud native SDN for Kubernetes @romanaproject
Network policy workflow
Kubernetes master
Kubernetes APIURLs
New URLs for this resource type, per
namespace
romana.io A cloud native SDN for Kubernetes @romanaproject
Host
RomanaAgent
iptables
Host
RomanaAgent
iptables
Network policy workflow
Kubernetes master
RomanaK8S listener
Kubernetes API
Host
RomanaAgent
New Romana policy definition
URLs
Events streamed
through GET request
Some client
POST /…..{ new policy }
iptables
Demo
romana.io A cloud native SDN for Kubernetes @romanaproject
Conclusion
● Cloud native architectures simplify things● Need a cloud native SDN to enjoy benefits● Romana:
– Cloud native without compromises
– Native network performance
– Mostly static config: Solid network
– Very easy to work with and understand
● Easy to try:– Simple installers for Kubernetes and OpenStack
romana.io A cloud native SDN for Kubernetes @romanaproject
Thank you!
● Romana Links– http://romana.io - Project home
– http://romana.io/blog - Blog
– https://github.com/romana/romana - Sources
● Contact– @romanaproject - Twitter
– info@romana.io - Email
– https://romana.slack.com/ - Slack channel
● Kubernetes links– http://bit.ly/1RMVkrr - CNI spec
Appendix: Romana technical notes
romana.io A cloud native SDN for Kubernetes @romanaproject
Semantic and topological addressing
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9 8 7 6 5 4 3 2 1 0
0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1
10
Network prefix bitsThe network prefix. In this example, we are using the 10/8
address space.
6
Host ID Segment IDWe currently
store tenant ID in upper bits of segment ID.
4 67
Endpoint ID
Widths are configurable, don't have to use byte boundaries.
romana.io A cloud native SDN for Kubernetes @romanaproject
Segment and tenant bits
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9 8 7 6 5 4 3 2 1 0
0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1
10
Network prefix bits
6
Host ID Segment ID
4 67
Endpoint ID
Encode thetenant ID
romana.io A cloud native SDN for Kubernetes @romanaproject
Romana: Traffic segmentation
● Tenant traffic separated:– Tenants don't get whole CIDR prefix or L2 domain
– But fully isolated from other tenants' traffic
● Tenants can define segments:– Like tiers, provide isolation and policies
● Use segment and tenant bits in IP addresses:– Apply policies (iptables) based on that
– Segments can stretch across hosts
romana.io A cloud native SDN for Kubernetes @romanaproject
Host BHost A
Allowing traffic within tenant
10.0.0.5 10.1.0.12
iptables:check src/dst addrs“tenant/segment bits
must match”
Src: 10.0.0.5Dst: 10.1.0.12
Same tenant/segment bits
romana.io A cloud native SDN for Kubernetes @romanaproject
Host BHost A
Isolating tenant traffic: Default
10.0.0.5 10.1.128.9
iptables:check src/dst addrs“tenant/segment bits
must match”
Src: 10.0.0.5Dst: 10.1.128.9
Different tenant/segment bits
Differenttenant
romana.io A cloud native SDN for Kubernetes @romanaproject
Host BHost A
Apply network policy between segments (full isolation as default)
10.0.0.5 10.1.1.9
iptables:Does policy chain
exist?Otherwise: DROP
Src: 10.0.0.5Dst: 10.1.1.9
Same tenant, different segment
policy-chain:From segment 0?Protocol TCP?To port 80?