The Journey of a Packet Through the Linux Network Stack plus hints on Lab 9

AssumptionsIP version 4Codes are from Kernel 2.6.9.ELIdeas are similar

Linux High-Level Network Stack

Interface to users

TCP/UDP/IP etc

Queue for device

Image from http://affix.sourceforge.net/affix-doc/c190.html

Receiving a Packet (Device)Network card receives a frame

Driver handles the interrupt

issues an interrupt

Frame RAMAllocates sk_buff (called skb)Frame skb

Aside: sk_buff (skbuff.h)Generic buffer for all packetsPointers to skb are passed up/downCan be linked together

Transport Header(TCP/UDP/ICMP)Network Header(IPv4/v6/ARP)MAC HeaderRaw

sk_buff (cont.)struct sk_buff*nextstruct sk_buff*prevstruct sk_buff_head *liststruct sock*skunion {tcphdr; udphdr; }h;union {iph; ipv6h;arph;}nh;union {raw} mac;.Transport HeaderNetwork HeaderMAC HeaderDATA

sk_buff (cont.)Understanding Linux Network Internals, Christian Benvenuti

Receiving a Packet (Device)Driver (cont.)calls device independentcore/dev.c:netif_rx(skb)puts skb into CPU queueissues a soft interruptCPUcalls core/dev.c:net_rx_action()removes skb from CPU queuepasses to network layer e.g. ip/arpIn this case: IPv4 ipv4/ip_input.c:ip_rcv()

Receiving a Packet (IP)ip_input.c:ip_rcv()

checksLength >= IP Header (20 bytes)Version == 4ChecksumCheck length again

callsip_rcv_finish()

callsroute.c:ip_route_input()

Aside: Finish/Slow suffixDivision into two stages is commonUsually called slow

The first stage cacheThe second stagetable

Receiving a Packet (routing)ipv4/route.c:ip_route_input()

ipv4/route.c:ip_route_input_slow()

Destination == me?YESip_input.c:ip_local_deliver()NOCalls ip_route_input_slow()

Can forward?Forwarding enabled?Know route?NOSends ICMP

Forwarding a PacketForwarding is per-device basisReceiving device!Enable/Disable forwarding in Linux:/proc file system Kernelread/write normally (in most cases)

Forwarding a Packet (cont.)ipv4/ip_forward.c:ip_forward()

.... a few more callscore/dev.c:dev_queue_xmit()Default queue: priority FIFOsched/sch_generic.c:pfifo_fast_enqueue()Others: FIFO, Stochastic Fair Queuing, etc.

IP TTL > 1YESDecreases TTLNOSends ICMP

Priority Based Output Schedulingpfifo_fast_enqueue()Again, per-device basisQueue Discipline (Qdisc: pkt_sched.c)Not exactly a priority queueUses three queues (bands)0 interactive 1 best effort 2 bulkPriority is based on IP Type of Service (TOS)Normal IP packet 1 best effort

Queue Discipline: Qdischttp://linux-ip.net/articles/Traffic-Control-HOWTO/classless-qdisc.html

Mapping IP ToS to QueueIP ToS: PPPDTRCXPPP PrecedenceLinux = ignoreCisco = Policy-Based Routing (PBR)D Minimizes DelayT Maximizes ThroughputR Maximizes ReliabilityC Minimizes CostX Reserved

Mapping IP ToS to Queue (cont.)pfifo_fast_enqueue() maps IP ToS to one of three queuesIP ToS: PPPDTRCXMapping array: prior2band

* Linux priority != band

IP ToSLinux PriorityBand0x0010x2120x4020x6020x8210xA220xC200xE200x10610x12610x14610x16610x18410x1A410x1C410x1E41

Queue SelectionMapping arrayBand 0 (first in Qdisc)Change bandsch_generic.c

Queue Selection (cont.)Kernel 2.6.9.ELQdiscsk_buff_headband 0sk_buff_headband 1sk_buff_headband 2list = ((struct sk_buff_head*)qdiscdata+prior2band[skb->priority&TC_PRIOR_MAX]

Sending Out a Packetpfifo_fast_dequeue()Removes the oldest packet from the highest priority bandThe packet that was just enqueued!Passes it to the device driver

Lab 9 Part 1&2SetupDestinationLinux Router(Your HDD)Virtual 1Virtual 2Bottleneck link: 10Mbps

Lab 9 Part 2Default: no IP forwardingEnable IP forwarding /proc/Only one routerDefault route on destination

Lab 9 Part 2DestinationLinux Router(Your Linux)Virtual 1Virtual 2Bottleneck link: 10Mbpsping echoRoute???ping reply

Lab 9 Part 3ScenarioDestinationLinux Router(Your Linux)Virtual 1Virtual 210MbpsTCPUDP

Lab 9 Part 3 (cont.)Problem with TCP v.s. UDP?TCP is too niceProposed solution: Modify kernel TCP higher priority

Lab 9 Part 4Goal: compile the modified kernelPrint out TCP/UDP when sending or forwarding a packet/proc/sys/kernel/printkStart up with the new kernel!Press any key on boot OS listSelect 2.6.9

Lab 9 Part 5Goal: change the kernel schedulingIdea: place TCP in the higher priority bandpfifo_fast_enqueue()Default IP ToSChange it to TCP v.s. UDP (+others)Options: UDP++ or TCP--Do NOT change IP ToS!

Lab 9 Part 5 (cont.)UDPTCP

Lab 9 Part 5 (cont.)

Lab 9 Part 5 (cont.)Remember: take printk() out!boot into 2.6.9enable forwardingWhat happen? Compare to Part 2?

*