ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting...
Transcript of ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting...
![Page 2: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/2.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
• Introduction
• Installation
• Configuration
• Tuning
• Use cases
Overview
![Page 3: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/3.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
• Open source packet processing framework for Linux.
• Originally (2003) designed to accelerate packet capture on commodity hardware, using patched drivers and in-kernel filtering.
• Today it supports almost all Intel adapters with kernel-bypass zero-copy drivers and almost all FPGAs capture adapters.
PF_RING
![Page 4: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/4.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
PF_RING's Main Features
• PF_RING consists of:
• Kernel module (pf_ring.ko)
• Userspace library (libpfring)
• Userspace modules implementing multi-vendor support
• Patched libpcap for legacy applications
PF_RING.ko
libpfring
Kernel
Userspace
PF_RING App
NIC
PCAP-Over-PF_RING
modules
PCAP App
![Page 5: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/5.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
PF_RING.ko
PF_RING lib
• Standard kernel drivers, NAPI polling.
• 1-copy by the NIC into kernel buffers (DMA).
• 1-copy by the PF_RING kernel module into memory-map’ed memory.
Standard Drivers
Kernel
Userspace
App
mmap()
1-copy
NIC
![Page 6: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/6.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
PF_RING ZC
• Userspace drivers for Intel cards, kernel is bypassed.
• 1-copy by the NIC into userspace memory (DMA).
• Packets are read directly by the application in zero-copy.
PF_RING ZC Drivers
NIC
Kernel
Userspace
App
DMA
0-copy
zc:eth1
![Page 7: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/7.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
• PF_RING ZC is not just a zero-copy driver, it provides a flexible API for creating full zero-copy processing patterns using 3 simple building blocks:
‣ Queue ‣ Hw Device Queue
‣ Sw SPSC Queue
‣ Pool: DMA buffers resource.
‣ Worker: execution unit able to aggregate traffic from M ingress queues and distribute it to N generic egress queues using custom functions.
PF_RING ZC API
Worker
Pool
![Page 8: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/8.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
PF_RING ZC API - zbalance Example
1/10G
Core 0 Core 1 Core 2 Core 3
Linux
1/10G
PF_RING
ZC
ConsumerThread 1
PF_RING
ZC
ConsumerThread 2
PF_RING
ZC
Sw Distribution
PF_RING
ZC
ConsumerThread 0
PoolPool Pool
Worker
![Page 9: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/9.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
• Code for aggregation and load-balancing using ZC:
PF_RING ZC API - zbalance code
1 zc = pfring_zc_create_cluster(ID, MTU, MAX_BUFFERS, NULL);
2 for (i = 0; i < num_devices; i++)
3 inzq[i] = pfring_zc_open_device(zc, devices[i], rx_only);
4 for (i = 0; i < num_slaves; i++)
5 outzq[i] = pfring_zc_create_queue(zc, QUEUE_LEN);
6 zw = pfring_zc_run_balancer(inzq, outzq, num_devices,
num_slaves, NULL, NULL, !wait_for_packet, core_id);
![Page 10: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/10.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
• Currently PF_RING natively supports the following vendors (1/10/40/100 Gbit)
• PF_RING-based applications transparently select the module by means of the interface name. Example:
• pfcount -i eth1 [Vanilla Linux adapter]
• pfcount -i zc:eth1 [Intel ZC drivers]
• pfcount -i nt:1 [Napatech]
• pfcount -i myri:1 [Myricom]
• pfcount -i exanic:0 [Exablaze]
FPGAs Support
![Page 11: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/11.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Many modules, single API.
Kernel
ZCIntel NIC
StandardDrivers
Standard NICNapatech
Card
ring buffer(packet copy)
NAPI
Napatech modZC mod
libpfring
pf_ring mod
Napatech lib
App
0-copy (DMA) FPGA
App App
zc:eth1eth0 nt:0
ZCDrivers
pf_ring.ko
Stack mod
App
stack:eth2
NetworkStack
MyricomCard
Myricom mod
SNF lib
FPGA
App
myri:0
App
n2disk mod
timeline:/storage
IndexPCAP
![Page 12: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/12.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Overview
• Introduction
• Installation
• Configuration
• Tuning
• Use cases
![Page 13: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/13.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
• Two options for installing PF_RING:
• Source Code (GitHub)
• Packages
‣ Stable
‣ Dev (aka “nightly builds”)
Installation
![Page 14: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/14.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
• Download
# git clone https://github.com/ntop/PF_RING.git
• Installation:
# cd PF_RING/kernel
# make && make install
# cd ../userland
# make && make install
• ZC drivers installation (optional):
# cd PF_RING/drivers/intel/<model>/<model>-<version>-zc/src
# make && make install
• Support for FPGAs (Napatech, Myricom, etc) is automatically enabled if drivers are installed.
Installation - Source Code
![Page 15: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/15.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
• CentOS/Debian/Ubuntu stable/devel repositories at http://packages.ntop.org
• Installation:
# wget http://apt.ntop.org/16.04/all/apt-ntop.deb
# dpkg -i apt-ntop.deb
# apt-get clean all
# apt-get update
# apt-get install pfring
• ZC drivers installation (optional):
# apt-get install pfring-drivers-zc-dkms
• Support for FPGAs (Napatech, Myricom, etc) is already there.
Installation - Packages
![Page 16: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/16.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Overview
• Introduction
• Installation
• Configuration
• Optimisation
• Use cases
![Page 17: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/17.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Loading PF_RING
• If you compiled from source code:
# cd PF_RING/kernel
# insmod ./pf_ring.ko
• If you are using packages:
# tree /etc/pf_ring/
|-- pf_ring.conf
`-- pf_ring.start
# /etc/init.d/pf_ring start
![Page 18: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/18.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
• ZC drivers are available for almost all Intel cards based on e1000e, igb, ixgbe, i40e, fm10k
• ZC needs hugepages for memory allocation, the pf_ring init script takes care of reserving them.
• A ZC interface acts as a standard interface (e.g. you can set an IP on ethX) until you open it using the “zc:” prefix (e.g. zc:ethX).
Loading ZC Drivers
![Page 19: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/19.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
• If you compiled from source code:
# cd PF_RING/drivers/intel/<model>/<model>-<version>-zc/src
# ./load_driver.sh
• In essence the script loads hugepages and dependencies and load the module with:
# insmod <model>.ko RSS=1,1 [other options]
• You can check that the ZC driver is actually running with:
# cat /proc/net/pf_ring/dev/eth1/info | grep ZC
Polling Mode: ZC/NAPI
Loading ZC Drivers
![Page 20: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/20.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
• If you are using packages (ixgbe driver in this example):
# tree /etc/pf_ring/
|-- hugepages.conf
|-- pf_ring.conf
|-- pf_ring.start
`-- zc
`-- ixgbe
|-- ixgbe.conf
`-- ixgbe.start
• Where:
# cat /etc/pf_ring/hugepages.conf
node=0 hugepagenumber=1024
# cat /etc/pf_ring/zc/ixgbe/ixgbe.conf
RSS=1,1
Loading ZC Drivers
![Page 21: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/21.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
RSS
Network Card
Core 0 Core 1 Core 2 Core 3
Queue 0 Queue 1 Queue 2 Queue 3
CPU
• RSS distributes the load across the specified number of RX queues based on an hash function which is IP-based (or IP/Port-based in case of TCP)
![Page 22: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/22.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
RSS
• Set the number of RSS queues using the insmod option or ethtool:
# ethtool --set-channels eth1 combined 4 # cat /proc/net/pf_ring/dev/eth1/info | grep Queues TX Queues: 4 RX Queues: 4
• In order to open a specific interface queue, you have to specify the queue ID using the "@<ID>" suffix.
# tcpdump -i zc:eth1@0
Note: when using ZC, “zc:eth1” is the same as “zc:eth1@0”! This happens because ZC is a kernel-bypass technology, there is no abstraction (queues aggregation) provided by the kernel.
![Page 23: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/23.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Indirection Table
• Destination queue is selected in combination with an indirection table:
queue = indirection_table[rss_hash(packet)]
• It is possible to configure the indirection table using ethtool by simply applying weights to each RX queue.
![Page 24: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/24.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Indirection Table
# ethtool --set-channels eth1 combined 4 # ethtool -x eth1 RX flow hash indirection table for eth1 with 4 RX ring(s): 0: 0 1 2 3 0 1 2 3 8: 0 1 2 3 0 1 2 3 16: 0 1 2 3 0 1 2 3 24: 0 1 2 3 0 1 2 3 32: 0 1 2 3 0 1 2 3 40: 0 1 2 3 0 1 2 3 48: 0 1 2 3 0 1 2 3 56: 0 1 2 3 0 1 2 3 64: 0 1 2 3 0 1 2 3 72: 0 1 2 3 0 1 2 3 80: 0 1 2 3 0 1 2 3 88: 0 1 2 3 0 1 2 3 96: 0 1 2 3 0 1 2 3 104: 0 1 2 3 0 1 2 3 112: 0 1 2 3 0 1 2 3 120: 0 1 2 3 0 1 2 3
destination queue ID
hash
![Page 25: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/25.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Indirection Table
# ethtool -X eth1 weight 1 0 0 0 # ethtool -x eth1 RX flow hash indirection table for eth1 with 4 RX ring(s): 0: 0 0 0 0 0 0 0 0 8: 0 0 0 0 0 0 0 0 16: 0 0 0 0 0 0 0 0 24: 0 0 0 0 0 0 0 0 32: 0 0 0 0 0 0 0 0 40: 0 0 0 0 0 0 0 0 48: 0 0 0 0 0 0 0 0 56: 0 0 0 0 0 0 0 0 64: 0 0 0 0 0 0 0 0 72: 0 0 0 0 0 0 0 0 80: 0 0 0 0 0 0 0 0 88: 0 0 0 0 0 0 0 0 96: 0 0 0 0 0 0 0 0 104: 0 0 0 0 0 0 0 0 112: 0 0 0 0 0 0 0 0 120: 0 0 0 0 0 0 0 0
hash
destination queue ID
![Page 26: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/26.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Overview
• Introduction
• Installation
• Configuration
• Tuning
• Use cases
![Page 27: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/27.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Xeon Architecture
PCIe PCIe
![Page 28: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/28.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
PCIe PCIe
QPI
• QPI (Quick Path Interconnect) is the bus that interconnects the nodes of a NUMA system.
• QPI is used for moving data between nodes when accessing remote memory or PCIe devices. It also carries cache coherency traffic.
![Page 29: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/29.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
PCIe PCIe
Memory
• Each CPU has its local memory directly attached.
• Accessing remote memory is slow as data flows through the QPI, which has lower bandwidth and adds latency.
E5-2687WV4 9.6 GT/s QPI76.8 GB/s RAM DDR4 2400QPI latency: hundreds of nanosec
Example:8.0 GT/s QPI - bandwidth 32 GiB/s ~32 GB/s
![Page 30: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/30.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
PCIe PCIe
PCIe
• Each node has its dedicated PCIe lanes.
• Plug the Network Card (and the RAID Controller) to the right slot reading the motherboard manual.
![Page 31: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/31.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
PCIe PCIe
Memory Channels
• Multi-channel memory increases data transfer rate between memory and memory controller. You can use n2membenchmark as benchmark tool.
• Check how many channels your CPU supports and use at least as many memory modules as the number of channels (check dmidecode).
![Page 32: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/32.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
PCIe PCIe
CPU Cores
• CPU pinning of a process/thread to a core is important to isolate processing and improve performance.
• In most cases dedicating a physical core (pay attention to hyper-threading) to each thread is the best choice for optimal performance.
![Page 33: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/33.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Core Affinity
• All our applications natively support CPU pinning, e.g.:
# nprobe -h | grep affinity [--cpu-affinity|-4] <CPU/Core Id> | Binds this process to the specified CPU/Core
• When not supported, you can use external tools:
# taskset -c 1 tcpdump -i eth1
![Page 34: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/34.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
NUMA Affinity
• You can check your NUMA-related hw configuration with:
• lstopo
• numactl --hardware
• Configuring CPU pinning, usually the application allocates memory to the correct NUMA node, if this is not the case you can use external tools:
# numactl --membind=0 --cpunodebind=0 tcpdump -i zc:eth1
• You can check your QPI bandwidth with:
# numactl --membind=0 --cpunodebind=1 n2membenchmark
![Page 35: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/35.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
PF_RING ZC Driver NUMA Affinity
• PF_RING ZC drivers allocate data structures (RX/TX ring) in memory, setting NUMA affinity is important. You can do that at insmod:
# insmod <model>.ko RSS=1,1,1,1 numa_cpu_affinity=0,0,8,8
• Or if you are using packages:
# cat /etc/pf_ring/zc/ixgbe/ixgbe.conf
RSS=1,1,1,1 numa_cpu_affinity=0,0,8,8
![Page 36: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/36.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Traffic Recording - Wrong Configuration
PCIe PCIe
1. NIC to memory
2. Processing in memory
3. Memory to Storage
![Page 37: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/37.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Traffic Recording - Correct Configuration
PCIe PCIe
![Page 38: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/38.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Overview
• Introduction
• Installation
• Configuration
• Tuning
• Use cases
![Page 39: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/39.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
RSS Load Balancing
Core 0 Core 1 Core 2 Core 3
Linux
PF_RING
ZC
ConsumerThread 1
PF_RING
ZC
ConsumerThread 2
PF_RING
ZC
ConsumerThread 3
PF_RING
ZC
ConsumerThread 0
Network Card
![Page 40: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/40.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
RSS: When it can be used
• Flow-based traffic analysis (multi-threaded or multi-process) and all the applications where Divide and Conquer strategy is applicable.
• Examples:
• nProbe (Netflow probe)
• nProbe Cento
• Suricata
• Bro
![Page 41: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/41.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
RSS: nProbe Example
• nProbe instances example with 4 RSS queues:
# nprobe -i zc:eth1@0 --cpu-affinity 0 [other options]
# nprobe -i zc:eth1@1 --cpu-affinity 1 [other options]
# nprobe -i zc:eth1@2 --cpu-affinity 2 [other options]
# nprobe -i zc:eth1@3 --cpu-affinity 3 [other options]
![Page 42: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/42.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
RSS: Bro Example
• Bro node.cfg example with 8 RSS queues:
# [worker-1]
type=worker
host=10.0.0.1
interface=zc:eth1
lb_method=pf_ring
lb_procs=8
pin_cpus=0,1,2,3,4,5,6,7
This is expanded into zc:eth1@0 .. zc:eth1@7
![Page 43: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/43.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
RSS: When it can NOT be used
• Applications where packets order has to be preserved (also across flows), especially if there is no hw timestamping.
• For example in n2disk (traffic recording) we have to keep the original order for packets dumped on disk.
![Page 44: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/44.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
ZC Load Balancing (zbalance_ipc)
1/10G
Core 0 Core 1 Core 2 Core 3
Linux
1/10G
PF_RING
ZC
ConsumerApp B T2
PF_RING
ZC
Consumer App A
PF_RING
ZC
Sw Distribution
PF_RING
ZC
ConsumerApp B T0
Worker
![Page 45: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/45.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
ZC Load Balancing: When it is useful
• When RSS is not available or not flexible enough (with ZC you can build your distribution function/hash)
• When you need to send the same traffic to multiple applications (fan-out) while using zero-copy
• When you need to aggregate from multiple ports and then distribute
![Page 46: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/46.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
ZC Load Balancing - example
• zbalance_ipc is an example of multi-process load balancing application:
# zbalance_ipc -i zc:eth1,zc:eth2 -c 99 -n 1,2 -m 1 -g 0
• Consumer applications example:
# taskset -c 1 tcpdump -i zc:99@0
# nprobe -i zc:99@1 --cpu-affinity 2 [other options]
# nprobe -i zc:99@2 --cpu-affinity 3 [other options]
Ingress Interfaces ZC ID Egress Queues
Hash Type
CPU Core
![Page 47: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/47.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
ZC Load Balancing and Bro
• Bro node.cfg example with 8 ZC queues:
# [worker-1]
type=worker
host=10.0.0.1
interface=zc:99
lb_method=pf_ring
lb_procs=8
pin_cpus=0,1,2,3,4,5,6,7
This is expanded into zc:99@0 .. zc:99@7
![Page 48: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/48.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Other processing patterns
Linux
Core 0 Core 1 Core 2 Core 3
PF_RING ZC
App B (e.g. IPS)
PF_RING ZC
App A(e.g. DDoS)
PF_RING
ZC
PacketDispatcher
1/10G
PF_RING
ZC
Packet Forwarder
1/10G
• Using the ZC API you can create any multithreaded or multi-process processing pattern. Pipeline example:
![Page 49: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/49.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
ZC & Virtualisation: PCI Passthrough
• Any hypervisor is supported: KVM, VMWare (Direct I/O), Xen, etc.
PF_RING ZC
NIC
Host
App
DMA0-copy
zc:eth1
NIC
Kernel
Userspace
VM
PCI Passthrough
![Page 50: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/50.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
ZC & Virtualisation: Host to VM (KVM)
Core 0 Core 1 Core 2 Core 3
KVM
PF_RING ZC
App(e.g. IPS)
PF_RING
ZC
PacketForwarder
1/10G
PF_RING
ZC
Packet Forwarder
1/10G
(Host) $ zpipeline_ipc -i zc:eth2,0 -o zc:eth3,1 -n 2 -c 99 -r 0 -t 2 -Q /tmp/qmp0
(VM) $ zbounce_ipc -c 99 -i 0 -o 1 -u
![Page 51: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/51.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Stack Injection
• ZC is a Kernel-Bypass technology: what if we want to forward some traffic to the Linux Stack?
NetworkStack
PF_RING ZC
Kernel
Userspace
App
DMA
0-copy
stack:eth1 zc:eth1
NIC
pfrin
g_se
nd()
pfrin
g_re
cv()
NIC
![Page 52: ntop Users Group Meeting - High Performance Network ... · PDF filentop Users Group Meeting ... Network Stack Myricom Card Myricom mod SNF lib FPGA App ... # cd PF_RING/drivers/intel//--zc/src](https://reader036.fdocuments.us/reader036/viewer/2022081401/5ab2ed5c7f8b9ac3348dc61a/html5/thumbnails/52.jpg)
Arnhem, Netherlands • October 17, 2016
meeting
Thank you!