MultiPath TCP : Linux Kernel implementation - Linux Plumbers
Transcript of MultiPath TCP : Linux Kernel implementation - Linux Plumbers
![Page 1: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/1.jpg)
MultiPath TCP : Linux Kernelimplementation
Presenter: Christoph PaaschIP Networking LabUCLouvain, Belgium
August 28, 2012
http://multipath-tcp.org
IP Networking Lab http://multipath-tcp.org 1 / 29
![Page 2: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/2.jpg)
Networks are becoming Multipath
Mobile devices can connect to the Internet via differentinterfaces
3G
WiFi
Internet
IP Networking Lab http://multipath-tcp.org 2 / 29
![Page 3: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/3.jpg)
Networks are becoming Multipath
Mobile devices can connect to the Internet via differentinterfaces
3G
WiFi
Internet
IP Networking Lab http://multipath-tcp.org 2 / 29
![Page 4: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/4.jpg)
However, TCP does not support this
Smartphones have to restart their data-transfer when movingaway from the WiFi access-point.
IP Networking Lab http://multipath-tcp.org 3 / 29
3G
WiFi
Internet
![Page 5: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/5.jpg)
However, TCP does not support this
Smartphones have to restart their data-transfer when movingaway from the WiFi access-point.
IP Networking Lab http://multipath-tcp.org 3 / 29
3G
WiFi
Internet
![Page 6: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/6.jpg)
However, TCP does not support this
Smartphones have to restart their data-transfer when movingaway from the WiFi access-point.
IP Networking Lab http://multipath-tcp.org 3 / 29
3G
WiFi
Internet
![Page 7: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/7.jpg)
Networks are becoming Multipath
Data-centers have a large redundant infrastructure
IP Networking Lab http://multipath-tcp.org 4 / 29
A B C
![Page 8: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/8.jpg)
Networks are becoming Multipath
Data-centers have a large redundant infrastructure
IP Networking Lab http://multipath-tcp.org 4 / 29
A B C
![Page 9: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/9.jpg)
However, TCP is suboptimal
Collisions in data-center reduce the bandwidth and result insuboptimal load-balancing
IP Networking Lab http://multipath-tcp.org 5 / 29
A B C
![Page 10: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/10.jpg)
However, TCP is suboptimal
Collisions in data-center reduce the bandwidth and result insuboptimal load-balancing
IP Networking Lab http://multipath-tcp.org 5 / 29
A B C
![Page 11: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/11.jpg)
However, TCP is suboptimal
Collisions in data-center reduce the bandwidth and result insuboptimal load-balancing
IP Networking Lab http://multipath-tcp.org 5 / 29
A B C
![Page 12: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/12.jpg)
However, TCP is suboptimal
Collisions in data-center reduce the bandwidth and result insuboptimal load-balancing
IP Networking Lab http://multipath-tcp.org 5 / 29
A B C
![Page 13: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/13.jpg)
Mismatch between the multipath networkand the single-path transport protocol.
IP Networking Lab http://multipath-tcp.org 6 / 29
![Page 14: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/14.jpg)
MultiPath TCP
Runs with unmodified applications
Works over today’s Internet
IPv4/IPv6 are both supported (even simultaneously)
IP Networking Lab http://multipath-tcp.org 7 / 29
![Page 15: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/15.jpg)
MultiPath TCP
IP Networking Lab http://multipath-tcp.org 8 / 29
WiFi - IP: A
3G - IP: B
Internet
IP: S
![Page 16: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/16.jpg)
MultiPath TCP
IP Networking Lab http://multipath-tcp.org 8 / 29
WiFi - IP: A
3G - IP: B
IP: S
![Page 17: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/17.jpg)
MultiPath TCP
Is the server MPTCP-capable?
IP Networking Lab http://multipath-tcp.org 8 / 29
SrcIP: ADstIP: S
SYNMP_CAPABLE : ID12
![Page 18: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/18.jpg)
MultiPath TCP
Is the server MPTCP-capable?
IP Networking Lab http://multipath-tcp.org 8 / 29
SrcIP: SDstIP: A
SYN + ACKMP_CAPABLE: ID23
![Page 19: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/19.jpg)
MultiPath TCP
Sending Data - Naiv approach
IP Networking Lab http://multipath-tcp.org 8 / 29
SrcIP: S DstIP: ASeqNo: 1
SrcIP: S DstIP: ASeqNo: 2
![Page 20: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/20.jpg)
MultiPath TCP
Sending Data - Naiv approach
IP Networking Lab http://multipath-tcp.org 8 / 29
SrcIP: S DstIP: ASeqNo: 1
SrcIP: S DstIP: ASeqNo: 2
SrcIP: S DstIP: ASeqNo: 3
![Page 21: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/21.jpg)
MultiPath TCP
Sending Data - Naiv approach
IP Networking Lab http://multipath-tcp.org 8 / 29
SrcIP: S DstIP: ASeqNo: 1
SrcIP: S DstIP: ASeqNo: 2
SrcIP: S DstIP: ASeqNo: 3
![Page 22: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/22.jpg)
MultiPath TCP
Sending Data - Naiv approach
IP Networking Lab http://multipath-tcp.org 8 / 29
SrcIP: S DstIP: ASeqNo: 1
SrcIP: S DstIP: ASeqNo: 2
SrcIP: S DstIP: ASeqNo: 3
NATFirewalls
![Page 23: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/23.jpg)
MultiPath TCP
Establish separate subflows
IP Networking Lab http://multipath-tcp.org 8 / 29
SrcIP: BDstIP: S
SYNMP_JOIN : ID23
![Page 24: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/24.jpg)
MultiPath TCP
Establish separate subflows
IP Networking Lab http://multipath-tcp.org 8 / 29
SrcIP: SDstIP: B
SYN + ACKMP_JOIN
![Page 25: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/25.jpg)
MultiPath TCP
Establish separate subflows
IP Networking Lab http://multipath-tcp.org 8 / 29
![Page 26: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/26.jpg)
MultiPath TCP
Establish separate subflows
IP Networking Lab http://multipath-tcp.org 8 / 29
write_seq
snd_cwnd
rcv_nxt
Subflow 1
write_seq
snd_cwnd
rcv_nxt
Subflow 2
![Page 27: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/27.jpg)
MultiPath TCP
Sending Data
IP Networking Lab http://multipath-tcp.org 8 / 29
write_seq
snd_cwnd
rcv_nxt
Subflow 1
write_seq
snd_cwnd
rcv_nxt
Subflow 21 2 3 4 5
write(...);
![Page 28: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/28.jpg)
MultiPath TCP
Sending Data
IP Networking Lab http://multipath-tcp.org 8 / 29
SubSeq: 102DataSeq: 2
SubSeq: 2001DataSeq: 3
SubSeq: 101DataSeq: 1
write_seq
snd_cwnd
rcv_nxt
Subflow 1
write_seq
snd_cwnd
rcv_nxt
Subflow 2
write_seq
rcv_nxt
MPTCP-level
![Page 29: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/29.jpg)
MultiPath TCP
Both subflows can be used simultaneously.
Addresses are advertised with ADD_ADDR and removed byREMOVE_ADDR.
Subflows can be dynamically added and removed during thelifetime of the connection.
IP Networking Lab http://multipath-tcp.org 9 / 29
![Page 30: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/30.jpg)
MultiPath TCP
Linux Kernel Implementation
Available at http://multipath-tcp.org
IP Networking Lab http://multipath-tcp.org 10 / 29
![Page 31: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/31.jpg)
Establishing first subflow
Exchanged Messages
SrcIP: ADstIP: S
SYNMP_CAPABLE : ID12
IP Networking Lab http://multipath-tcp.org 11 / 29
![Page 32: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/32.jpg)
Establishing first subflow
High-Level Kernel design - Client Side
standard Socket API
Transport Layer
Application Layer
Network Layer
Connection EstablishmentIs the Peer MPTCP-Capable?
TCPsubflow
structtcp_sock
socket(AF_INET, SOCK_STREAM, 0);connect(...);
IP Networking Lab http://multipath-tcp.org 11 / 29
![Page 33: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/33.jpg)
Establishing first subflow
In-depth call-stack - Client Sidesocket(AF_INET, SOCK_STREAM, 0);connect(...);
tcp_v4_connect(...);
tcp_connect(...);
tcp_connect_init(...);Generate a new identifier for this session
create SYN
tcp_transmit_skb(...);tcp_syn_options(...);
mptcp_syn_options(...);
tcp_options_write(...);
mptcp_options_write(...);
pass packet to IP-stack
IP Networking Lab http://multipath-tcp.org 11 / 29
![Page 34: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/34.jpg)
Establishing first subflow
In-depth call-stack - Server Side
tcp_v4_rcv(...);
tcp_v4_conn_request(...);
tcp_openreq_init(...);
* Store recv ident. in mptcp_request_sock* Generate locally unique ident.
SYN
create the request-socket
tcp_v4_send_synack(...);
mptcp_synack_options(...);
SYN/ACK
IP Networking Lab http://multipath-tcp.org 12 / 29
![Page 35: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/35.jpg)
Establishing first subflow
Exchanged Messages
SrcIP: SDstIP: A
SYN + ACKMP_CAPABLE: ID23
IP Networking Lab http://multipath-tcp.org 12 / 29
![Page 36: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/36.jpg)
Establishing first subflow
High-Level Kernel design - Client Side
standard Socket API
Transport Layer
Application Layer
Network Layer
MultiPath TCPstruct tcp_sock
Creating the MPTCP-leveland linking the application to this socket.TCP
subflow
structtcp_sock
IP Networking Lab http://multipath-tcp.org 12 / 29
![Page 37: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/37.jpg)
Establishing first subflow
In-depth call-stack - Client Side
tcp_v4_rcv(...);
tcp_v4_do_rcv(...);
tcp_rcv_state_process(...);
tcp_rcv_synsent_state_process(...);
mptcp_alloc_mpcb(...);create the MPTCP-level socket
mptcp_add_sock(...);attach init. subflow to MPTCP
SYN/ACK
IP Networking Lab http://multipath-tcp.org 12 / 29
![Page 38: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/38.jpg)
Establishing first subflow
Exchanged Messages
ACK
IP Networking Lab http://multipath-tcp.org 12 / 29
![Page 39: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/39.jpg)
Establishing first subflow
In-depth call-stack - Server Side
tcp_v4_hnd_req(...);tcp_v4_do_rcv(...);tcp_v4_rcv(...);
tcp_check_req(...);
ACK
tcp_v4_syn_recv_sock(...);
Creates the initial subflow-socket
mptcp_check_req_master(...);Creates the MPTCP-level socketInitializes everything for MPTCP
Handles the third ackof the 3-way Handshake
IP Networking Lab http://multipath-tcp.org 12 / 29
![Page 40: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/40.jpg)
Establishing first subflow
High-Level Kernel design
standard Socket API
Transport Layer
Application Layer
Network Layer
MultiPath TCP
TCPsubflow
structtcp_sock
struct tcp_sock
IP Networking Lab http://multipath-tcp.org 12 / 29
![Page 41: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/41.jpg)
Establishing additional subflows
Exchanged Messages
SrcIP: BDstIP: S
SYNMP_JOIN : ID23
IP Networking Lab http://multipath-tcp.org 13 / 29
![Page 42: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/42.jpg)
Establishing additional subflows
High-Level Kernel design - Client Side
standard Socket API
Transport Layer
Application Layer
Network Layer
MultiPath TCP
An additional subflowis being createdTCP
subflow
structtcp_sock
TCPsubflow
structtcp_sock
struct tcp_sock
IP Networking Lab http://multipath-tcp.org 13 / 29
![Page 43: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/43.jpg)
Establishing additional subflows
In-depth call-stack - Client Sidemptcp_init4_subsockets(...);
connect(...);
mptcp_add_sock(...);
Creates a new IPv4 subflow
Adds the MP_JOIN
tcp_v4_connect(...);
tcp_connect(...);
SYN
Attach subflow to MPTCP-level
IP Networking Lab http://multipath-tcp.org 13 / 29
![Page 44: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/44.jpg)
Establishing additional subflows
In-depth call-stack - Server Side
tcp_v4_rcv(...);
SYN
tcp_v4_send_synack(...);
mptcp_lookup_join(...);
SYN/ACK
Look for the MP_JOIN in a SYNAnd see if we know this identifier
mptcp_v4_do_rcv(...);
mptcp_v4_join_request(...);
Similar to tcp_v4_conn_request()
IP Networking Lab http://multipath-tcp.org 14 / 29
![Page 45: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/45.jpg)
Establishing additional subflows
Exchanged Messages
SrcIP: SDstIP: B
SYN + ACKMP_JOIN
IP Networking Lab http://multipath-tcp.org 14 / 29
![Page 46: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/46.jpg)
Establishing additional subflows
Exchanged Messages
ACKMP_JOIN
IP Networking Lab http://multipath-tcp.org 14 / 29
![Page 47: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/47.jpg)
Establishing additional subflows
In-depth call-stack - Server Side
tcp_v4_rcv(...);
ACK
mptcp_check_req_child(...);
mptcp_syn_recv_sock(...);Look for a request-sock
mptcp_v4_do_rcv(...);
tcp_check_req(...);
mptcp_add_sock(...);Attach subflow to MPTCPInitialize subflow for MPTCP
IP Networking Lab http://multipath-tcp.org 14 / 29
![Page 48: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/48.jpg)
Sending Data
Exchanged Messages
SubSeq: 102DataSeq: 2
SubSeq: 2001DataSeq: 3
SubSeq: 101DataSeq: 1
write_seq
snd_cwnd
rcv_nxt
Subflow 1
write_seq
snd_cwnd
rcv_nxt
Subflow 2
write_seq
rcv_nxt
MPTCP-level
IP Networking Lab http://multipath-tcp.org 15 / 29
![Page 49: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/49.jpg)
Sending Data
High-Level Kernel design
standard Socket API
Transport Layer
Application Layer
Network Layer
struct tcp_sock MultiPath TCPsend-queue
write(sock, &buf, size);
21 3 4 5
TCPsubflow
structtcp_sock
TCPsubflow
structtcp_sock
IP Networking Lab http://multipath-tcp.org 15 / 29
![Page 50: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/50.jpg)
Sending Data
High-Level Kernel design
standard Socket API
Transport Layer
Application Layer
Network Layer
MultiPath TCP
TCPsubflow
structtcp_sock
TCPsubflow
structtcp_sock
MultiPath TCP Scheduler
send-queue
write(sock, &buf, size);
21 3 4 5
struct tcp_sock
IP Networking Lab http://multipath-tcp.org 15 / 29
![Page 51: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/51.jpg)
Sending Data
High-Level Kernel design
standard Socket API
Transport Layer
Application Layer
Network Layer
MultiPath TCPsend-queue
send-queue send-queue
21 3 4 5
21 3 4 5
write(sock, &buf, size);
TCPsubflow
structtcp_sock
TCPsubflow
structtcp_sock
MultiPath TCP Scheduler
struct tcp_sock
IP Networking Lab http://multipath-tcp.org 15 / 29
![Page 52: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/52.jpg)
Receiving Data
Packets can be reordered at the data-level due to delay-differences.
IP Networking Lab http://multipath-tcp.org 16 / 29
SubSeq: 1DataSeq: 1000
SubSeq: 200DataSeq: 1001
SubSeq: 2DataSeq: 1002
![Page 53: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/53.jpg)
Receiving Data
Packets can be reordered at the data-level due to delay-differences.
IP Networking Lab http://multipath-tcp.org 16 / 29
1000
SubSeq: 200DataSeq: 1001
receive-queue
MPTCP-Level
SubSeq: 2DataSeq: 1002
![Page 54: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/54.jpg)
Receiving Data
Packets can be reordered at the data-level due to delay-differences.
IP Networking Lab http://multipath-tcp.org 16 / 29
1000
SubSeq: 200DataSeq: 1001
out-of-order queue
1002
receive-queue
MPTCP-Level
![Page 55: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/55.jpg)
Receiving Data
A loss at the subflow-level (or network-reordering) can also causereordering at the subflow-level
IP Networking Lab http://multipath-tcp.org 17 / 29
SubSeq: 3DataSeq: 1003
SubSeq: 4DataSeq: 1004
![Page 56: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/56.jpg)
Receiving Data
A loss at the subflow-level (or network-reordering) can also causereordering at the subflow-level
IP Networking Lab http://multipath-tcp.org 17 / 29
SubSeq: 4DataSeq: 1004 Packet Dropped
![Page 57: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/57.jpg)
Receiving Data
Subflow-level out-of-order queues are necessary to handle theretransmission at the subflow-level
IP Networking Lab http://multipath-tcp.org 18 / 29
SubSeq: 4DataSeq: 1004
out-of-order queueSubflow-Level
![Page 58: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/58.jpg)
Receiving Data
High-Level Kernel design
standard Socket API
Transport Layer
Application Layer
Network Layer
MultiPath TCPofo-queuereceive-queue
ofo-queue ofo-queue
read(sock, &buf, size);
TCPsubflow
structtcp_sock
TCPsubflow
structtcp_sock
struct tcp_sock
IP Networking Lab http://multipath-tcp.org 19 / 29
![Page 59: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/59.jpg)
MultiPath TCP
Design Challenges
IP Networking Lab http://multipath-tcp.org 20 / 29
![Page 60: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/60.jpg)
Writing the Data-sequence number in the TCP-options
Naive approach
s t r u c t t c p s k b cb {u32 seq ; /∗ S t a r t i n g sequence number ∗/
[ . . . ]#i f d e f CONFIG MPTCP
u32 da t a s eq ;u32 end da t a s eq ;u32 da ta ack ;
[ . . . ]#end i f}
Writing data-seq in tcp options write
Increased tcp skb cb by 24 bytes.
IP Networking Lab http://multipath-tcp.org 21 / 29
![Page 61: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/61.jpg)
Writing the Data-sequence number in the TCP-options
Our solution
Inside the MPTCP-scheduler write the data-seq on top of thepayload, before calling tcp transmit skb.
struct sk_buff
char *data;
Payload
Data-seq
No more increase of tcp skb cb.
IP Networking Lab http://multipath-tcp.org 22 / 29
![Page 62: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/62.jpg)
Creating the MPTCP-level on the client-side.
Current Design
standard Socket API
Transport Layer
Application Layer
Network Layer
Connection EstablishmentIs the Peer MPTCP-Capable?
TCPsubflow
structtcp_sock
socket(AF_INET, SOCK_STREAM, 0);connect(...);
IP Networking Lab http://multipath-tcp.org 23 / 29
![Page 63: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/63.jpg)
Creating the MPTCP-level on the client-side.
Current Design
standard Socket API
Transport Layer
Application Layer
Network Layer
MultiPath TCPstruct tcp_sock
Creating the MPTCP-leveland linking the application to this socket.TCP
subflow
structtcp_sock
IP Networking Lab http://multipath-tcp.org 23 / 29
![Page 64: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/64.jpg)
Creating the MPTCP-level on the client-side.
Current Design
IP Networking Lab http://multipath-tcp.org 23 / 29
struct socket
struct sock *sk;
Application-level
MPTCP-level
TCP subflow-level
struct tcp_sock
struct mptcp_cb *mpcb;
struct mptcp_tcp_sock *mptcp;
![Page 65: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/65.jpg)
Creating the MPTCP-level on the client-side.
Current Design
IP Networking Lab http://multipath-tcp.org 23 / 29
struct socket
struct sock *sk;
Application-level
MPTCP-level
TCP subflow-level
struct mptcp_cb
struct tcp_sock
MPTCP specific variables
Pointers to subsockets
sk_clone()
struct tcp_sock
struct mptcp_cb *mpcb;
struct mptcp_tcp_sock *mptcp;
![Page 66: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/66.jpg)
Creating the MPTCP-level on the client-side.
Current Design
IP Networking Lab http://multipath-tcp.org 23 / 29
struct socket
struct sock *sk;
Application-level
MPTCP-level
TCP subflow-level
struct mptcp_cb
struct tcp_sock
struct mptcp_tcp_sock
MPTCP-specific fields
struct tcp_sock
struct mptcp_cb *mpcb;
struct mptcp_tcp_sock *mptcp;
![Page 67: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/67.jpg)
Creating the MPTCP-level on the client-side.
Current Design
IP Networking Lab http://multipath-tcp.org 23 / 29
struct socket
struct sock *sk;
Application-level
MPTCP-level
TCP subflow-level
struct mptcp_cb
struct tcp_sock
struct mptcp_tcp_sock
MPTCP-specific fields
struct tcp_sock
struct mptcp_cb *mpcb;
struct mptcp_tcp_sock *mptcp;
Change sk-pointer
![Page 68: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/68.jpg)
Creating the MPTCP-level on the client-side.
Current Design
Problems, if the application does a system-call on the socket,before the reception of the SYN+ACK
Fix: Wait for the SYN+ACK. E.g., tcp_sendmsg:
/∗ Wait f o r a connec t i on to f i n i s h . ∗/i f ( (1 << sk−>s k s t a t e ) & ˜(TCPF ESTABLISHED |
TCPF CLOSE WAIT) )i f ( ( e r r = s k s t r e am wa i t c o n n e c t ( sk , &t imeo ) )
!= 0)goto d o e r r o r ;
We need to do this in all functions that take a lock on thesocket! tcp_recvmsg, tcp_splice_read, ip_setsockopt,ip_get_sockopt, tcp_ioctl, . . . and many more
IP Networking Lab http://multipath-tcp.org 24 / 29
![Page 69: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/69.jpg)
Creating the MPTCP-level on the client-side.
Upcoming Design
standard Socket API
Transport Layer
Application Layer
Network Layer
struct tcp_sock
Connection EstablishmentIs the Peer MPTCP-Capable?
IP Networking Lab http://multipath-tcp.org 25 / 29
![Page 70: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/70.jpg)
Creating the MPTCP-level on the client-side.
Upcoming Design
standard Socket API
Transport Layer
Application Layer
Network Layer
MultiPath TCPstruct tcp_sock
Creating the subsocketand linking it to theMPTCP-levelTCP
subflow
structtcp_sock
IP Networking Lab http://multipath-tcp.org 25 / 29
![Page 71: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/71.jpg)
Creating the MPTCP-level on the client-side.
Upcoming Design
IP Networking Lab http://multipath-tcp.org 25 / 29
struct socket
struct sock *sk;
Application-level
MPTCP-level
TCP subflow-level
struct tcp_sock
struct mptcp_cb *mpcb;
struct mptcp_tcp_sock *mptcp;
![Page 72: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/72.jpg)
Creating the MPTCP-level on the client-side.
Upcoming Design
IP Networking Lab http://multipath-tcp.org 25 / 29
struct socket
struct sock *sk;
Application-level
MPTCP-level
TCP subflow-level
sk_clone()
struct tcp_sock
struct mptcp_cb *mpcb;
struct mptcp_tcp_sock *mptcp;
struct tcp_sock
struct mptcp_cb *mpcb;
struct mptcp_tcp_sock *mptcp;
![Page 73: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/73.jpg)
Creating the MPTCP-level on the client-side.
Upcoming Design
IP Networking Lab http://multipath-tcp.org 25 / 29
struct socket
struct sock *sk;
Application-level
MPTCP-level
TCP subflow-level
struct mptcp_tcp_sock
Per socket MPTCP-specific
fields
struct mptcp_cb
MPTCP specific variables
Pointers to subsockets
struct tcp_sock
struct mptcp_cb *mpcb;
struct mptcp_tcp_sock *mptcp;
struct tcp_sock
struct mptcp_cb *mpcb;
struct mptcp_tcp_sock *mptcp;
![Page 74: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/74.jpg)
Handling socket options
Questions
Lots of socket options in the TCP/IP stack
Some are for the MPTCP-level (SO SNDBUF ), some shouldget passed onto all other subflows (IP TTL)
This requires a lot of changes in TCP unrelated functions(e.g., do_ip_setsockopt)
How could we handle this?
IP Networking Lab http://multipath-tcp.org 26 / 29
![Page 75: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/75.jpg)
Still lots of changes to the TCP-stack
Questions
We have a lot of:
i f ( t c p s k ( sk )−>mpc) {DO SOME MPTCP STUFF
} e l s e {DO USUAL TCP STUFF
}
IP Networking Lab http://multipath-tcp.org 27 / 29
![Page 76: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/76.jpg)
Submitting MPTCP upstream???
∼ 10000 lines of code
Tightly integrated in the TCP-stackMore work to do:
Cleanup - better separate MPTCP from TCPSome missing featuresSupport TSOSupport NET DMA. . .
How to split the patch in small pieces?
IP Networking Lab http://multipath-tcp.org 28 / 29
![Page 77: MultiPath TCP : Linux Kernel implementation - Linux Plumbers](https://reader031.fdocuments.us/reader031/viewer/2022022322/6216526bc671f20ea95f37a5/html5/thumbnails/77.jpg)
Conclusion
Freely available at http://multipath-tcp.orgDownload it, try it out, contribute!
UCLouvain MPTCP-Team:Sebastien BarreChristoph PaaschGregory DetalFabien Duchene
Prof. Olivier Bonaventure
Thanks to our previous and present partners/contributors:
IP Networking Lab http://multipath-tcp.org 29 / 29