Post on 07-Jan-2022
Andreas la QuianteCNX1999 , …, CCIE , … CCONP2019::9
alq@nvidia.comHamburg, GermanyDate: 2020-SEP-16
OPEN NETWORKINGCUMULUS LINUX
73. DFN-Betriebstagung
2
ANDREAS
乐珼安安德烈亚斯
Danke für die Einladung!
3
AGENDA
乐珼安安德烈亚斯
16-September-2020Slot: 10:00 - 10:30 Uhr
Cumulus – Hardware offene Switch Betriebssysteme
Andreas la Quiante, Senior Systems Engineer, NVIDIA (Cumulus Networks)Cell: (+49 (0) 151535 76676)
4
01 - MOTIVATION
5
CONTROL AND DATA PLANE
Control-Plane
Data-Plane
乐珼安安德烈亚斯
Often:
48 x 1/10/25G
32 x 100G/200G/400G
6
OPTIONS … MANY OPTIONS
乐珼安安德烈亚斯
today…and we have only limited time…we need to focus!
7
Implementation Option 1
乐珼安安德烈亚斯
Open == Linux as a Bootloader / Open == Shell Access
Vendor
SDK
e.g. Vendor
Routing
Linux
Kernel
User Space
Kernel Space
Hardware
Offload
8
Implementation Option 2
乐珼安安德烈亚斯
Network Processor Vendor SDK
Vendor
SDK
Community
Routing
Linux
Kernel
Netlink
User Space
Kernel Space
Hardware
Offload
Use of the Linux Infrastructure
9
Implementation Option 2 (gute Virtualisierung)
乐珼安安德烈亚斯
NetworkProcessor Vendor SDK
Vendor
SDK
Community
Routing
Linux
Kernel
Netlink
Community
Routing
Linux
Kernel
VM
10
Implementation Option 3
乐珼安安德烈亚斯
Native Linux with Driver
Community
Routing
Linux
Kernel
Netlink
Features
https://github.com/Mellanox/mlxsw/wiki
User Space
Kernel Space
Hardware
Offload
driver
11
WE COVER TODAY: CUMULUS LINUX
乐珼安安德烈亚斯
Cumulus Linux
12
NETWORKING IS SIMILAR TO COOKING/DINING
乐珼安安德烈亚斯
Use Case 1 Use Case 2 Use Case 3
You need spices to
prepare different dishes
Like Cumulus Linux offering
different switch configurations
13
WHAT HAVE YOU OR COULD YOU CREATE?
Three examples:
乐珼安安德烈亚斯
Use Case 1 Use Case 2 Use Case 3
Research
HPC or Storage
(…vSphere 7.0)
RDMA over Converged Ethernet
Enterprise
Automation focus
DevOps
Linux Shop
Service Provider
K8S as a Service
Multi-Tenant with RoH
(EVPN-VXLAN)
14
MOTIVATION
Cumulus/Linux networking
offers for many,
especially Linux and DevOps customers,
Consistency and a positive community!
乐珼安安德烈亚斯
And allows extremely simple and native automation via common open source tools
efficient == clever
15
02 - PRODUCT
16
Products and Offerings backed by a great team
▪ Cumulus Linux 1.x
▪ Cumulus Linux 2.x
▪ Cumulus Linux 3.x ESR*
▪ Cumulus Linux 4.x
▪ NetQ 1.x
▪ NetQ 2.x
▪ NetQ 3.x
▪ Consulting Services
▪ Global Support Services
▪ Training and Education
▪ Documentation and an
active Community
*ESR == Extended Support Release
18
03 - ARCHITECTURE
19
Architecture – a personal view
乐珼安安德烈亚斯
www.onie.org
ifupdown2
vrf
vlan aware bridge
vxlan / evpn
neighbor mgr
quagga / frr
acl tool
portwd
switchd
NetQ Agent
PTM
…
ZTP
Image
(ZTP script
runs once)
20
Architecture - ideal for simulations
乐珼安安德烈亚斯
Control-Plane
Data-Plane
FRR
iproute2
mstpd
brctl
iptable
ebtable
ip6table
ifupdown2
Routing Tables ARP Table Bridge FDB Filter Tables
ethtool
cl-acltoollldpd
snmpd
swp1
Linux Kernel
NetQ
Agent
Soft
ware
only
part
21
simple simulation example
乐珼安安德烈亚斯
Server01 Server02
Leaf01 Leaf02
Spine01
22
Creation via .dot file
graph „Linux02“ {
"server01" [function="host" os="generic/ubuntu1804" ssh_user="cumulus" config="tc_files/cl.sh"]
"server02" [function="host" os="generic/ubuntu1804" ssh_user="cumulus" config="tc_files/cl.sh"]
"leaf01" [function="leaf" os="CumulusCommunity/cumulus-vx" version="4.1.1" ]
"leaf02" [function="leaf" os="CumulusCommunity/cumulus-vx" version="4.1.1" ]
"spine01" [function="spine" os="CumulusCommunity/cumulus-vx" version="4.1.1" ]
"server01":"eth1" -- "leaf01":"swp1"
"server02":"eth1" -- "leaf02":"spw1"
"spine01":"swp11" -- "leaf01":"swp11"
"spine01":"spw12" -- "leaf02":"swp12"
}
TopologyConverter.py
Ergebnis: Vagrant Datei
Provider: VB or KVM
air.cumulusnetworks.com
$ ssh -p xxxxx cumulus@air.cumulusnetworks.com
乐珼安安德烈亚斯
23
Single DC Design with OOB (Cumulus AIR)
https://air.cumulusnetworks.com/Login
乐珼安安德烈亚斯
Ansible EVPN Playbooks
(Production Ready Automation)
L2 EVPN
L3 Centralized Routing
L3 Decentralized Sym. Routing
NetQ
24
Architecture, physical Switch
乐珼安安德烈亚斯
Control-Plane
Data-Plane
FRR
iproute2
mstpd
brctl
iptable
ebtable
ip6table
ifupdown2
Routing Tables ARP Table Bridge FDB Filter Tables
ethtool
switchdcl-acltoollldpd
snmpd
swp1
swp1 (xe0)
Linux Kernel
Network
Prozessor
NetQ
Agent
25
Excurs Netlink Manager
乐珼安安德烈亚斯
NLMGR
Your SW
https://github.com/CumulusNetworks/python-nlmanager
Routing Tables
31
04 – USE-CASE EXAMPLES
32
DevOps, Linux and Automation
乐珼安安德烈亚斯
Automation focus
DevOps
Linux Shop • Open Source affine
Linux NOS
• Standard Orchestration Tools
in use or planned
• Leaf-Spine Fabric
with Overlays (EVPN-VxLAN)
33
DevOps, Linux and Automation
Automation ZTP
DHCP
(eth0)
OOB
Network
ZTP (runs once)
• User/Password
• Management VRF
• SSH Keys
• License File
Orchestration
• Ansible etc.
[t]
n Minutes
Up
and
Running
乐珼安安德烈亚斯
AM
SE
1 to 1000 nodes
34
Standard – Native - Automation
乐珼安安德烈亚斯
ServerSwitch
e/n/I
ifreload -aifupdown2
$ ifreload -a
+ an interface manager for scale, ease and non-disruption
35
The consistent way to manage
Many compute nodes (bare-metal, VM, …)
Some networking nodes, one open tool
or similar
乐珼安安德烈亚斯
36
Spices for Use-Cases
乐珼安安德烈亚斯
HPC or Storage
(…vSphere 7.0)
RDMA over
Converged Ethernet• BGP unnumbered
• Pause
• ECN
• ECN with VxLAN
37
BGP UNNUMBERED
Initiator Target
spine01
leaf01 leaf02 leaf03 leaf04
eth1 eth1
乐珼安安德烈亚斯
Layer 3 Data Center Fabric
Without the need to explicitly
Configure Layer 3 Addresses,
remote Layer 3 Addresses or
Remote ASNs!
BGP Configuration on leaf01
Router bgp 42000011
bgp router-it 10.0.0.11
neighbor swp51 interface remote-as external
39
LEAF-SPINE + OVERLAY + ECN
RoCEv1
RoCEv2
RoCEv2 & VXLAN
Initiator Target
spine02 spine03
leaf01 leaf02 leaf03 leaf04
eth1
Congestion
leads to setting
ECN in the (outer) IP Header
Mapping ECN from outer
to inner Header
Reacting on
ECN marking
Marking return traffic
eth1
VxLAN
乐珼安安德烈亚斯
spine01 spine04
40
Spices for Use-Cases
乐珼安安德烈亚斯
K8S as a Service
Multi-Tenant with RoH
(EVPN-VXLAN) • FRR, VRF, maybe even VTEPs
on switches and server
• Layer 2, Layer MLAG or
Layer 3 node connectivity
• Container running on CL
41
K8S
乐珼安安德烈亚斯
Compute (4.19.32)
Compute (4.19.32)
VxLAN
42
K8S as a Service (Financial Vertical)
乐珼安安德烈亚斯
Tenant A
K8S Maste/Minions
Tenant A FW
VxLAN VNIs:
Red outside
Green inside
Address-family l2vpn evpn
…
neighbor swp2 route-map FW out
route-map FW permit 1
match evpn vni 10020
match evpn vni 10021
C
C
C Container for
Provisioning
Partner IP
43
05 – SUMMARY
44
A DC FABRIC FOR ALL USE CASES
乐珼安安德烈亚斯
Compute ComputeStorage StorageAI AI
ONE FABRICAll use-cases All use-cases
45
EVPN – Layer 2 (CL 4.2 EVPN-MH) and Layer 3 (centralized, decentralized (both asym and sym)Multi-Tenant
Server01
VID: 10
VRF: LinuxAdmin
IP 172.31.10.1/24
swp1
bridge
VLAN 10
svi10
swp1
bridge
VLAN 10
Server02
VID: 10
VRF: LinuxAdmin
Server02‘
VID: 20
VRF: LinuxAdmin
swp
bridge
VLAN 20
svi20
VNI 10
VNI10010
Swp n
bridge
VLAN 30
svi30
VRF UnixAdmin
Server06
VID: 30
VRF: UnixAdmin
VNI 30
VNI 10030
cumulus@leaf01:mgmt:~$ net show evpn vni
VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs Tenant VRF
10 L2 vxlan10 2 4 0 LinuxAdmin
10010 L3 L3VNI_Linux 1 2 n/a LinuxAdmin
VNI 3020
乐珼安安德烈亚斯
IP 172.31.10.111/24
172.31.10.110
IP 172.31.10.2/24 IP 172.31.20.2/24
Vlan 1001 Vlan 1001
46
If you like one uniform data center and tool set
EVPN-VxLAN
THANK YOU
48
NEXT STEPS
• Test Drive (theory chapters + hands-on lab, free of charge)
• Try it yourself (download VX or whole Vagrant lab)
• Use Cumulus Air with 3 pre-build EVPN setups
• You need Linux networking for your project?
• You need a SME for your meeting?
• A future proof/reliable infrastructure is key?
• You like or need to automate?
乐珼安安德烈亚斯
Call !
49
READING SUGGESTIONS
乐珼安安德烈亚斯
50
Reference Material
乐珼安安德烈亚斯 https://cumulusnetworks.com/learn/web-scale-networking-resources/cheatsheets/
One skill-set to rule them all