Evolution of TCP / UDP

2

User Datagram Protocol (UDP)

• Strengths– Lightweight Protocol– No overhead

• Weaknesses– No reliability in UDP

CS423 - Cotter

Client Server

rqst

resp

3

Transmission Control Protocol (TCP)• Strengths

– Reliable Data Transmission– Stream Protocol

• Weaknesses– Connection Overhead– Transaction time:

• 2 RTT +• Server Processing time (SPT) +• 1RTT for each additional data

packet• Connection resources held until

any possible lost packets have expired ( 2 * RTTmax = 240 sec)

CS423 - Cotter

Client Server

syn

syn, ack

ack

dataack

ack

fin

fin

data

4

Minimum TCP Transaction

• Protocol allows sending data with syn– Not generally done– Must hold data until

3WHS complete

• Minimum transmission– 5 packets– 2 RTT + SPT

CS423 - Cotter

Client Server

syn, fin,data

syn, ack

ack, fin

ack

ack, findata

5

The Problem:• Some applications (particularly Web apps) need both reliability

and speed– Many connections (especially page loads) need only a few data packets– Early studies showed that median HTTP reply between 1770 and 958

bytes – May have many connections per session.– Overhead of connection (syn, syn-ack, ack) increases the transaction

time– Google study showed 4 to 40% improvement in page load time

possible– Because ports cannot be reused for 240 sec, maximum transaction rate

between a client and server is ~64000 ports/ 240 sec = 267 trans/sec– Problem is most pronounced when RTT is large

CS423 - Cotter

6

One Solution:TCP for Transactions (T/TCP)

• Originally proposed in rfc 1644 (1994)

• Include data in original syn packet.

• Maintain a Connection Counter (cc) in server that is passed by client to server as TCP option. Server uses this value to validate client, bypassing the need for 3-way handshake (3WHS)

• Because server can now validate each connection, connection delay reduced from 240 to 12 sec.

CS423 - Cotter

Client Server

syn, cc,data, fin

syn, ack, cc, data, fin

ack

7

T/TCP Initial Connection Setup

• Protocol requires setting up the connection count between each client and the server. Initial 3WHS includes CC request.

• Each succeeding connection uses CC and avoids 3WHS

• Protocol requires that CC increment monotonically.

CS423 - Cotter

T/TCP Protocol

• Initial 3WHS • Succeeding data calls

CS423 - Cotter 8

Client Server

syn, ack,CCecho

fin, ack

Data, fin

ack

ack

syn, CCnew,data

ack

Client Server

syn, cc,data, fin

syn, ack, cc, data, fin

ack

9

T/TCP Problems !

• For most implementations, server initializes CC at 1 and increments by 1 for each new connection. By sniffing the connection, it is easy to anticipate and spoof the CC. Even without access to the path, picking a large number will usually succeed (CC is 32 bit number)

• This makes it very easy for an attacker to implement a syn attack (DoS). It is made more effective since server must store the data associated with the syn, using up more server resources with each new packet.

• Protocol moved to historical status in 2011.

CS423 - Cotter

10

TCP Fast Open

• Researched originally at Google in 2011. – Published at CoNEXT 2011– Objective was to improve web server performance

• Offered as IETF draft first in 2012.– Currently in draft 08 (expires August, 2014)

• Implemented in Linux (since Kernel 3.6/3.7)

CS423 - Cotter

11

TCP Fast Open

• Use initial connection to establish relationship.– Client requests TCP fast open cookie and sends data in original

syn packet– Server creates cookie and returns to client in syn, ack packet.

Cookie is an encrypted hash of client IP address and other data.– Client ack packet completes 3WHS.– Server processes client request and returns response.

• Subsequent requests do not need 3WHS– Client request includes server cookie, data– Server returns syn, ack– Server returns response– Client completes handshake.

CS423 - Cotter

12

Fast Open Initial connect

CS423 - Cotter

13

Fast Open Subsequent Packets

CS423 - Cotter

14

Linux Implementation• Requires current kernel (at least 3.7 for client and server)• Requires setting /proc/sys/net/ipv4/tcp_fastopen

– Enable TCP Fast Open feature (draft-ietf-tcpm-fastopen) to send data in the opening SYN packet. To use this feature, the client application must use sendmsg() or sendto() with MSG_FASTOPEN flag rather than connect() to perform a TCP handshake automatically. The values (bitmap) are

1: Enables sending data in the opening SYN on the client w/ MSG_FASTOPEN. 2: Enables TCP Fast Open on the server side, i.e., allowing data in a SYN

packet to be accepted and passed to the application before 3-way hand shake finishes. 4: Send data in the opening SYN regardless of cookie availability and without

a cookie option. 0x100: Accept SYN data w/o validating the cookie. 0x200: Accept data-in-SYN w/o any cookie option present. 0x400/0x800: Enable Fast Open on all listeners regardless of the

TCP_FASTOPEN socket option. The two different flags designate two different ways of setting max_qlen without the TCP_FASTOPEN socket option.

Default: 1 Note that the client & server side Fast Open flags (1 and 2 respectively) must be also enabled before the rest of flags can take effect.

CS423 - Cotter

15

Linux TCP Fast Open client (TFO_client.cpp)void UDPecho(const char *host, const char *service) {

char buf[BUFFSIZE][LINELEN+1] = {"This is the first message", "This is the

second message", "This is the third message", "This is the fourth message",

"This is the last message" }; char inbuf[LINELEN +1];int s, nchars, rchars; /* socket

descriptor, character counters */:

for (index = 0; index < BUFFSIZE; index++) {

s = socket(AF_INET, SOCK_STREAM, 0);:nchars = strlen(buf[index]);CS423 - Cotter

16

Linux TCP Fast Open client (TFO_client.cpp)

status = sendto(s, &buf[index], nchars, MSG_FASTOPEN, (const struct sockaddr *) &sin, sizeof(sin));:

cout << "We just sent a datagram" << endl;rchars = recvfrom(s, inbuf, LINELEN, 0, NULL, NULL);

:inbuf[rchars] = '\0'; //Add a null termination

:cout << "We got back the following " << rchars << " characters: "

<< inbuf << endl;close (s);sleep(1);

} /* end of while */}/* end of UDPecho () */

CS423 - Cotter

17

Linux TCP Fast Open server (TFO_server.cpp)

int main(int argc, char *argv[]) {int qlen = 5;

:sock = passiveTCP(service, qlen);ans = setsockopt(sock, IPPROTO_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));while (keep_looping == YES) {

ssock = accept (sock, (sockaddr *)&fsin, (socklen_t *)&fsinsize);sprintf (log_msg, "Just got a request from %s\n", inet_ntoa(fsin.sin_addr));logData(log_msg);cc = recv (ssock, buf, LINELEN, 0);buf[cc] = '\0';sprintf (log_msg, "Msg: %s\n", buf);logData(log_msg);strcat (buf, " answer");send (ssock, buf, cc +7, 0);memset (buf, 0, LINELEN);close (ssock);

}//end of while loop}CS423 - Cotter

18

Wireshark Trace – Initial connect (1)cookie request

CS423 - Cotter

19

Wireshark Trace – Initial connect (2)cookie

CS423 - Cotter

20

Wireshark Trace – Initial connect (3)data

CS423 - Cotter

21

Wireshark Trace – Initial connect (4)ack

CS423 - Cotter

22

Wireshark Trace – Initial connect (4)response

CS423 - Cotter

23

Wireshark Trace – Next connect (1)syn, cookie, data

CS423 - Cotter

24

Wireshark Trace – Next connect (2)syn,ack

CS423 - Cotter

25

Wireshark Trace – Next connect (3)response

CS423 - Cotter

26

Wireshark Trace – Next connect (4)fin, ack

CS423 - Cotter

27

Fast Open Client side Output[bob@localhost client]$ g++ -o tfoc TFO_client.cpp[bob@localhost client]$ ./tfoc 192.168.1.25 23456Our target server is at address 192.168.1.25Enter data to send...We just sent a datagramWe got back the following 32 characters: This is the first message answerWe just sent a datagramWe got back the following 33 characters: This is the second message answerWe just sent a datagramWe got back the following 32 characters: This is the third message answerWe just sent a datagramWe got back the following 33 characters: This is the fourth message answerWe just sent a datagramWe got back the following 31 characters: This is the last message answer[bob@localhost client]$ CS423 - Cotter

28

Fast Open Server Log### Starting Server on port 23456 at Wed Apr 2 11:26:17 2014 Just got a request from 192.168.1.117Msg: This is the first messageJust got a request from 192.168.1.117Msg: This is the first messageJust got a request from 192.168.1.117Msg: This is the second messageJust got a request from 192.168.1.117Msg: This is the third messageJust got a request from 192.168.1.117Msg: This is the fourth messageJust got a request from 192.168.1.117Msg: This is the last message

CS423 - Cotter

29

Extended Fast Open Test• Build Standard TCP Client

– Send 100 packets, receive 100 responses

• Build TCP Fast Open Client– Send 100 packets, receive 100 responses

• Test Conditions– Client and server on adjacent machines on same LAN– Same code structure, same data sent and received.

• Compare Results– How many Packets required for each

• Standard: 1000 packets• TFO: 801

– How much time required for each?• Standard: .173333 sec• TFO: .119198 sec

CS423 - Cotter

30

Summary

• Network Protocols continue to evolve• TCP Fast Open offers the speed of UDP (after

the first connection) with the reliability of TCP.• Protocol is still in experimental stage.

CS423 - Cotter

31

References• TCP/IP Illustrated Vol 3, Richard Stevens

– Chapters 1-12 (T/TCP)• Original TCP Fast Open Paper

– http://conferences.sigcomm.org/co-next/2011/papers/1569470463.pdf– http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/

37517.pdf• IETF draft RFC

– https://datatracker.ietf.org/doc/draft-ietf-tcpm-fastopen/• lwn.net (Linux Weekly News?) Article

– https://lwn.net/Articles/508865/• Packet Pushers Article

– http://packetpushers.net/tcp-fast-curious-a-look-at-tcp-fast-open/• Admin Magazine Article

– http://www.admin-magazine.com/Articles/TCP-Fast-Open• Linux Kernel documentation

– https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txtCS423 - Cotter