ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of...
Transcript of ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of...
![Page 1: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/1.jpg)
ClickOS and the Art of Network Function Virtualization
Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto Bifulco*, Vladimir Olteanu§, Michio Honda*, Felipe Huici*
* NEC Labs Europe, Heidelberg, Germany § University Politehnica of Bucharest
[email protected], [email protected]
(NSDI 2014 Paper)
![Page 2: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/2.jpg)
The Idealized Network
Physical
Datalink
Network
Transport
Application
Physical
Datalink
Network
Transport
Application
Physical
Datalink
Network
Physical
Datalink
Page 2
![Page 3: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/3.jpg)
A Middlebox World
Page 3
carrier-grade NAT
load balancer
DPI QoE monitor
ad insertion
BRAS
session border controller
transcoder
WAN accelerator
DDoS protection
firewall
IDS
![Page 4: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/4.jpg)
Hardware Middleboxes - Drawbacks
▐ Expensive equipment/power costs
▐ Difficult to add new features (vendor lock-in) ▐ Difficult to manage ▐ Cannot be scaled on demand (peak planning)
Page 4
![Page 5: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/5.jpg)
Page 5
Shifting Middlebox Processing to Software
▐ Can share the same hardware across multiple users/tenants ▐ Reduced equipment/power costs through consolidation ▐ Safe to try new features on a operational network/platform ▐ But can it be built using commodity hardware while still
achieving high performance?
▐ ClickOS: tiny Xen-based virtual machine that runs Click
![Page 6: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/6.jpg)
From Thought to Reality - Requirements
Page 6
30 msec boot times
ClickOS
5MB when running
provided by Xen
10Gb/s line rate* 45 µsec delay
* for most packet sizes
provided by Click
▐ Fast Instantiation ▐ Small footprint ▐ Isolation ▐ Performance ▐ Flexibility
![Page 7: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/7.jpg)
What's ClickOS ? domU
paravirt
apps
guest OS
ClickOS
paravirt
Click
mini OS
Page 7
▐ Work consisted of: l Build system to create ClickOS images (5 MB in size) l Emulating a Click control plane over MiniOS/Xen l Reducing boot times (roughly 30 milliseconds) l Optimizations to the data plane (10 Gb/s for almost all pkt sizes) l Implementation of a wide range of middleboxes
![Page 8: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/8.jpg)
netback
packet size (bytes)
10 Gbit/s rate
64 14.88 Mp/s
128 8.4 Mp/s
256 4.5 Mp/s
512 2.3 Mp/s
1024 1.2 Mp/s
1500 810 Kp/s
Performance analysis
Page 8
Driver Domain (or Dom 0) ClickOS Domain
Xen bus/store
Event channel
netfront
Xen ring API (data)
NW driver OVS
300* Kp/s 350 Kp/s 225 Kp/s * - maximum-sized packets
vif
Click
ToDevice
FromDevice
![Page 9: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/9.jpg)
Performance analysis
Page 9
▐ Copying packets between guests greatly
affects packet I/O (1)
▐ Packet metadata allocations (2) ▐ Backend switch is slow (3) ▐ MiniOS netfront not as good as Linux
netback
Driver Domain (or Dom 0) ClickOS Domain
Xen bus/store
Event channel
netfront
Xen ring API
NW driver OVS
vif
Click
ToDevice
FromDevice
772 ns (1) ~600 ns (2) ~3.4 us (3)
![Page 10: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/10.jpg)
Optimizing Network I/O – Backend Switch
Page 10
VALE netback
Driver Domain (or Dom 0) ClickOS Domain
netfront Xen bus/store
Event channel
Xen ring API (data)
NW driver (netmap mode)
port
Click
FromDevice
ToDevice
▐ Reuse Xen page permissions (frontend) ▐ Introduce VALE[1] as the backend switch ▐ Increase I/O requests batch size
OVS
[1] VALE, a switched ethernet for virtual machines, ACM CoNEXT'2012 Luigi Rizzo, Giuseppe Lettieri Universita di Pisa
![Page 11: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/11.jpg)
VALE
Optimizing Network I/O
Page 11
Driver Domain (or Dom 0) ClickOS Domain
netfront
NW driver Click
FromDevice
ToDevice
netback
Netmap API (data)
▐ Minimal memory requirements
– For max. throughput a guest only needs 2 MB of memory
▐ Breaks other (non-MiniOS) guests
– But we have implemented Linux netfront driver
slots KB (per ring)
# grants (per ring)
64 135 33
128 266 65
256 528 130
512 1056 259
1024 2117 516
2048 4231 1033
netback
port
Xen bus/store
Event channel
Xen ring API (data)
![Page 12: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/12.jpg)
ClickOS Prototype Overview
▐ Click changes are minimal ~600 LoC
▐ New toolstack for fast boot times ▐ Cross compile toolchain for MiniOS-based apps ▐ netback changes comprise ~500 LoC ▐ netfront (Linux/MiniOS) around ~600 LoC ▐ VALE switch extended to:
– Connect NIC ports and modular switching
Page 12
![Page 13: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/13.jpg)
EVALUATION
Page 13
![Page 14: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/14.jpg)
Experiments ▐ ClickOS Instantiation ▐ State reading/insertion performance ▐ Delay compared with other systems ▐ Memory footprint ▐ Switch performance for 1+ NICs ▐ ClickOS/MiniOS performance ▐ Chaining experiments ▐ Scalability over multiple guests ▐ Scalability over multiple NICs ▐ Implementation and evaluation of middleboxes ▐ Linux Performance
Page 14
![Page 15: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/15.jpg)
ClickOS Base Performance
Intel Xeon E1220 4-core 3.2GHz (Sandy bridge) 16GB RAM, 1x Intel x520 10Gb/s NIC. One CPU core assigned to VMs, the rest to the Domain-0 Linux 3.6.10
Page 15
ClickOS Measurement Box
10Gb/s direct cable
![Page 16: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/16.jpg)
ClickOS Base TX Performance
Page 16
![Page 17: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/17.jpg)
ClickOS (virtualized) Middlebox Performance
Page 17
ClickOS Host 2 Host 1
10Gb/s direct cable 10Gb/s direct cable
Intel Xeon E1220 4-core 3.2GHz (Sandy bridge) 16GB RAM, 2x Intel x520 10Gb/s NIC. One CPU core assigned to Vms, 3 CPU cores Domain-0 Linux 3.6.10
![Page 18: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/18.jpg)
ClickOS (virtualized) Middlebox Performance
Page 18
![Page 19: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/19.jpg)
Linux Guest Performance
▐ Note that our Linux optimizations apply only to netmap-based applications
Page 19
![Page 20: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/20.jpg)
Conclusions
Page 20
▐ Virtual machines can do flexible high speed networking ▐ ClickOS: Tailor-made operating system for network processing
l Smaller is better: Low footprint is the key to heavy consolidation l Memory footprint: 5MB l Boot time: 30ms
▐ Future work: l Massive consolidation of VMs (thousands) l Improved Inter-VM communication for service chaining l Reactive VMs (e.g., per-flow)
![Page 21: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/21.jpg)
ClickOS Boot times
Page 21
30 milliseconds
220 milliseconds
![Page 22: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/22.jpg)
Scaling out – Multiple NICs/VMs
Intel Xeon E1650 6-core 3.2GHz, 16GB RAM, dual-port Intel x520 10Gb/s NIC. 3 cores assigned to VMs, 3 cores for dom0
Page 22
ClickOS Host 2
6x 10Gb/s direct cable 6x 10Gb/s direct cable
Host 1
![Page 23: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/23.jpg)
Scaling out – 100 VMs Aggregate Throughput
Intel Xeon E1650 6-core 3.2GHz, 16GB RAM, dual-port Intel x520 10Gb/s NIC. 3 cores assigned to VMs, 3 cores for dom0
Page 23
![Page 24: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/24.jpg)
ClickOS Delay vs. Other Systems
Page 24
![Page 25: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/25.jpg)
Towards Massive Server Consolidation
Filipe Manco, João Martins, Felipe Huici
{filipe.manco,joao.martins,felipe.huici}@neclab.eu
NEC Europe Ltd.
Xen Developer Summit 2014
![Page 26: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/26.jpg)
18 August 2014 26
The Super Fluid Cloud
● Target: remove barriers in current cloud deployments – Extremely flexible infrastructure – Milliseconds instantiation and migration of resources – Thousands of concurrent units running
● This would allow new use cases – On the fly deployment of middleboxes – Flash crowds – Energy consumption reduction – Your use case here...
![Page 27: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/27.jpg)
18 August 2014 27
Recent trend: specialized guests
● ClickOS, OSv, Mirage, Erlang on Xen, etc – Small memory footprints – Relatively fast boot times – Provide the basic functionality to make use cases a reality
● Our work focuses on ClickOS – Targets network processing using the Click modular router software
![Page 28: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/28.jpg)
18 August 2014 28
Wouldn't it be Nice if...
● Thousands of guests on a single server – Short-term target: 10K – Medium-term target: 100K
● Extremely fast domain creation, destruction and migration – Tens of milliseconds – Constant as number of guests increases
![Page 29: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/29.jpg)
18 August 2014 29
Experiment Setup
● Freshly installed Xen/Debian system – Xen 4.2 – Linux 3.6.10 – Debian squeeze
● Commodity server – 64 Cores @ 2.1GHz [4 x AMD Opteron 6376] – 128GB RAM DDR3 @ 1333MHz
![Page 30: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/30.jpg)
Xen and ClickOS Architecture
© NEC Corporation 2009
Xen
NIC
Dom0 (Linux)
Min
iOS
Click
ClickOS VM 1 ClickOS VM 2 ClickOS VM n
nw driver
netbackXenbus
Xen
storesw switch k
ernel
user-sp
ac e
netfrontXenbus
netfrontXenbus
netfrontXenbus
Clic
kO
SC
on
trol
Fro
mN
etfro
nt
To
Netfro
nt
middleboxconfig
Xen UI libraries
xl cosmos
SWIG
Pythontool
vif
TOOLSTACK
![Page 31: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/31.jpg)
18 August 2014 31
Baseline Test
Boot as many guests as possible before system breaks ● Using ClickOS guests
– 8 MB of RAM – 1 VIF
● Guests are mostly idle – Running arp responder configuration – Only arping guests to check they're working
![Page 32: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/32.jpg)
18 August 2014 32
Didn't Work Quite Well...
● Stopped test after 4K guests – Took ~ 5 days – Up to ~ 100 seconds for creation of last guest (normally ClickOS
boots in ~30 milliseconds) ● All the domains were running, but:
– Only first ~300 guests fully functional ● System got extremely slow
– Dom0 unusable
![Page 33: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/33.jpg)
18 August 2014 33
Domain Creation Time
![Page 34: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/34.jpg)
18 August 2014 34
Domain Creation Time
92 s
![Page 35: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/35.jpg)
18 August 2014 35
Two Types of Problems
● Hard limitations – Prevent guests from booting correctly – Only ~300 guests fully usable
● Performance limitations – Decreasing system performance – System unusable after just a few hundred guests
![Page 36: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/36.jpg)
HARD LIMITATIONS
© NEC Corporation 2009
![Page 37: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/37.jpg)
18 August 2014 37
Issues
● Cannot access guests' console – Only first ~300 guests have accessible console
● Guests' VIF is not created – Only first ~1300 guests have usable VIF
● Guests cannot access the Xenstore – Only first ~1300 guests have access to it
● The back-end switch doesn't provide enough ports – Only 1024 available
![Page 38: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/38.jpg)
18 August 2014 38
Number of File Descriptors
● xenconsoled opens 3 FD per guest – /dev/xenbus; /dev/ptmx; /dev/pts/<id>;
● Fix – Linux can easily handle > 300K FD – Tune fs.file-max; nofile ulimit;
![Page 39: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/39.jpg)
18 August 2014 39
Number of PTYs
● xenconsoled opens 1 PTY per guest ● Fix
– Linux can easily handle > 100K PTY – Tune kernel.tty.max
● Future – Only create PTY when user connects to console – This also reduces number of FD to 1 per guest
![Page 40: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/40.jpg)
18 August 2014 40
Number of Event Channels
● 3 Interdomain evtchn per guest – xenstore; console; VIF – 64bit Dom0: NR_EVTCHNS == 4096 – Dom0 runs out after ~1300 guests
● Fix – Upgrade to Xen 4.4 + Linux 3.14:
● NR_EVTCHNS == 128K
– Split services into stub domains
![Page 41: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/41.jpg)
18 August 2014 41
Number of IRQs
● Linux runs out of IRQs to map evtchn – Limited by NR_CPUS
● Fix – Build with: MAXSMP=y; NR_CPUS=4096 – NR_IRQS == 256K
![Page 42: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/42.jpg)
18 August 2014 42
vSwitch Ports
● Currently back-end switch supports up to few thousand ports – Linux bridge: 1K – Open vSwitch: 64K
● Workaround – Create multiple bridges
● Longer-term fix – Develop a purpose-built back-end switch
![Page 43: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/43.jpg)
18 August 2014 43
Summarizing
● Xen 4.4; Linux 3.14 ● fs.file-max; nofile ulimit ● kernel.tty.max ● MAXSMP=y; NR_CPUS=4096 ● Not yet fixed:
– Back-end switch ports
![Page 44: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/44.jpg)
PERFORMANCE LIMITATIONS
© NEC Corporation 2009
![Page 45: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/45.jpg)
18 August 2014 45
Issues
● Overall system becomes too slow – oxenstored
● CPU fully utilized after a few dozen guests – Xenconsoled
● CPU limited after ~ 2K guests
● Domain creation takes too long – Affects migration too
![Page 46: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/46.jpg)
18 August 2014 46
“Blind” optimizations
● 4 Core Dom0 – 1 core for oxenstored – 1 core for xenconsoled – 2 cores for remaining processes
● Pin all vCPUs to pCPUs ● Round robin remaining 60 cores for guests ● Put everything in a ramfs
![Page 47: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/47.jpg)
18 August 2014 47
Tools' Optimizations
● xl toolstack – Disable xl background process (xl create -e) – Disable memory ballooning on Dom0 – Never use domain name
● This causes xl to retrieve all guest names from the Xenstore – Use specialized VIF hotplug script – Don't retrieve domain list on creation [PATCH]
● oxenstored – Use more recent version of Xenstore from:
• https://github.com/mirage/ocaml-xenstore
![Page 48: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/48.jpg)
18 August 2014 48
Creation Times with Optimizations
![Page 49: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/49.jpg)
18 August 2014 49
Creation Times with Optimizations
2.3 s
![Page 50: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/50.jpg)
18 August 2014 50
How much better is it?
![Page 51: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/51.jpg)
18 August 2014 51
With Optimizations
● Improvement: system is still usable after 10K guests – Although domain creation time is far from ideal
● However... – xenstored still CPU heavy – xenconsoled still CPU heavy
![Page 52: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/52.jpg)
18 August 2014 52
xenconsoled
● Two major optimizations – Move from poll to epoll – On INTRODUCE_DOMAIN, search from last domid
● Avoid listing all existing domains
● CPU usage down to ~ 10% max.
![Page 53: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/53.jpg)
18 August 2014 53
What Bottlenecks Remain?
![Page 54: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/54.jpg)
18 August 2014 54
Domain Creation Breakdown
![Page 55: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/55.jpg)
18 August 2014 55
Let's Look at the Toolstack Again
● The domain creation process is too complex for our specialized VMs – Also makes the profiling really difficult and inaccurate – A lot of unnecessary Xenstore entries
● Some checks take a lot of time – Mainly checking for duplicate names
![Page 56: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/56.jpg)
18 August 2014 56
xcl: XenCtrl Light
● A very simplified toolstack ● Small abstraction on top of libxc (~600 LOC)
– Optimized for our use case ● Only boots PV and PVH domains ● Only supports VIFs
– Reduced Xenstore usage ● From 37 to 17 entries per guest ● Less Xenstore operations
– Doesn't check domain name
![Page 57: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/57.jpg)
18 August 2014 57
xl vs xcl
![Page 58: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/58.jpg)
18 August 2014 58
xl vs xcl
0.1 s
2.3 s
![Page 59: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/59.jpg)
18 August 2014 59
With xcl ● Much better ● But reducing the number of Xenstore entries is only a palliative
– Eventually the issue will come back as we increase the number of guests ● Xenstore remains a major bottleneck
![Page 60: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/60.jpg)
18 August 2014 60
lixs: LIghtweight XenStore
● Work in progress (about 1 month) ● Written from scratch but compatible with the Xenstore protocol ● Currently ~1800 LOC ● C++
![Page 61: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/61.jpg)
18 August 2014 61
lixs vs oxenstored
Cumulative time: ~ 11 min
Cumulative time: ~ 8 min
![Page 62: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/62.jpg)
18 August 2014 62
Breakdown with lixs
![Page 63: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/63.jpg)
18 August 2014 63
lixs: Future Work
● Optimize protocol – Make Xenstore more specialized – Avoid all possible listing operations
● Optimize implementation – Remove unix sockets – Generic storage backend
● std::map; noSQL DB; <your backend here>; ● 10K guests with std::map took 10m 3s ● 10K guests with boost::unordered_map took 7m 54s
![Page 64: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/64.jpg)
18 August 2014 64
Where are we?
● Usable system running 10K guests ● 10K guests actually working
– Although idle most of the time ● Lower domain creation times
– First domain: < 10ms – With 10K domains: < 100ms
– Recent test: 1,000 VMs running ICMP responder configuration, plus one running content cache (Minicache) – All 1,001 VMs work as expected!
![Page 65: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/65.jpg)
18 August 2014 65
Will it work? Can we reach 100K?
● There are no fundamental issues with Xen – But we only tested it up to 10K guests
● Xenstore protocol needs work – Make Xenstore more specialized – With 10K+ guests we need to avoid listings
![Page 66: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/66.jpg)
18 August 2014 66
Future work
● Improve lixs and Xenstore protocol ● Multi thousand-port vSwitch ● Have guests doing useful work ● Scheduling
– Number of guests much bigger than number of cores – With that many guests we'll have scheduling issues
● Reducing Memory Usage – Smaller image sizes – Share memory between guests booting same image
![Page 67: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/67.jpg)
18 August 2014 67
Xenstore Entries: xl vs xcl XL
1 = ""
vm = "/vm/2baefa82-612c-4e5b-a52d-396a91d5ad7b"
name = "proxy"
cpu = ""
0 = ""
availability = "online"
memory = ""
static-max = "8192"
target = "8193"
videoram = "-1"
device = ""
suspend = ""
event-channel = ""
vif = ""
0 = ""
backend = "/local/domain/0/backend/vif/46/0"
backend-id = "0"
state = "1"
handle = "0"
mac = "00:16:3e:32:ca:23"
1 = ""
backend = "/local/domain/0/backend/vif/46/1"
backend-id = "0"
state = "1"
handle = "1"
mac = "00:16:3e:2e:22:7c"
control = ""
shutdown = ""
platform-feature-multiprocessor-suspend = "1"
platform-feature-xs_reset_watches = "1"
data = ""
domid = "46"
store = ""
port = "1"
ring-ref = "3188551"
console = ""
backend = "/local/domain/0/backend/console/46/0"
backend-id = "0"
limit = "1048576"
type = "xenconsoled"
output = "pty"
tty = "/dev/pts/1"
port = "2"
ring-ref = "3188550"
XCL
1 = ""
control = ""
shutdown = ""
vm = "/vm/4c3f2a04-e39f-4ad8-9d7f-1b5556f02b34"
name = "proxy"
domid = "48"
console = ""
port = "2"
ring-ref = "3157830"
type = "xenconsoled"
tty = "/dev/pts/1"
device = ""
vif = ""
0 = ""
backend = "/local/domain/0/backend/vif/48/0"
backend-id = "0"
state = "1"
handle = "0"
mac = "00:00:00:00:00:00"
1 = ""
backend = "/local/domain/0/backend/vif/48/1"
backend-id = "0"
state = "1"
handle = "1"
mac = "00:00:00:00:00:00"
![Page 68: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/68.jpg)
18 August 2014 68
Number of grants
● 2 grants per domain – xenstore; xenconsole; – With v1: 512 grants per frame – DEFAULT_MAX_NR_GRANT_FRAMES == 32
● Maximum of (512 * 32) / 2 == 8K
● Fix – Boot xen with max_nr_grant_frames=512
● Up to 128K domains
![Page 69: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/69.jpg)
It's Open Source!
Page 69
Checkout l ClickOS, Backend Switch, Xen optimizations and more! l Tutorials l Better performance than listed in the papers!
![Page 70: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/70.jpg)
We are always looking for…
© NEC Corporation 2009
Interns Visiting researchers Collaborations
(and often full-time researchers)
Interested? [email protected]
![Page 71: ClickOS and the Art of Network Function Virtualization · 2019-06-14 · ClickOS and the Art of Network Function Virtualization Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Roberto](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fcdd6f0c71913d8113533/html5/thumbnails/71.jpg)