TCP Mobility/Splicing

TCP Mobility/Splicing

Francis Chang <[email protected]>

Systems Software Lab

OGI

The Papers

[3] Maltz, “TCP Splicing for Application Layer Proxy Performance”[4] Spatscheck, “Optimizing TCP Forwarder Performance

[2] Snoeren, “An End-to-End Approach to Host Mobility”

[1] Snoeren, “Fine-Grained Failover Using Connection Migration

Divided into two groups:

TCP Splicing TCP Mobility

TCP Splicing

Let’s first look at TCP Splicing…

TCP Splicing – The IssueDef’n: Split Connection Proxy -

A proxy machine is interposed between the server and the client machine in order to mediate the communication between them.

Client Proxy ServerTCP TCP

eg. HTTP cache, security firewall, encryption servers, SOCKS proxy

TCP Splicing – The Problem

Split Connection Proxies are slow!The proxy must maintain 2 TCP connections.

Client Server

Net1 Net2

IP IP

TCP TCP

Proxy

Unoptimized TCP Forwarder

Hardware:

Kernel:

Process:

TCP Splicing – The ProblemWhat’s wrong with split connection proxies?

The proxy must maintain 2 TCP connections. Needs a lot of memory & CPU processing

Performance is often CPU limited

Performance degradation due to lost packets (One connection must wait for the other to recover)

Potentially Violate the End-to-End semantics of TCP

TCP Splicing – The Ideal

Client Proxy ServerTCP TCP

How can we remove these problems from the proxy concept?

Ideally, we’d like to be able eliminate the middle-man.

TCP Splicing – The Ideal

Client ServerTCP

How can we remove these problems from the proxy concept?

Ideally, we’d like to be able eliminate the middle-man.

But there is no way to “join” 2 connections in TCP.

What’s the next best thing? Splicing!

TCP Splicing – The SolutionQ. What is TCP Splicing?

A. Joining 2 TCP connection so that the endpoints can communicate as peers.

Client Server

Net1 Net2

IP IP

TCP TCP

Proxy

Unoptimized TCP Forwarder

Hardware:

Kernel:

Process:

TCP Splicing – The SolutionQ. What is TCP Splicing?

A. Joining 2 TCP connection so that the endpoints can communicate as peers.

Client Server

Net1 Net2

IP IP

FWD

TCP Forwarder with Splicing

Hardware:

Kernel:

Process:

TCP Splicing – How it worksUse an IP forwarder, but munge the packets as they travel between the Server and Client

Simply have to renumber the IP src/dest addresses and ports, as well as the sequence numbers

Spatscheck[4] introduces the idea that if we carefully choose the TCP src/dest ports as well as initial sequence numbers, we don’t even have to touch the TCP portion of the packet

TCP Splicing – Trace

A SOCKS Packet Trace

TCP Splicing – Data

Now, let’s look at some data to see what TCPSplicing buys us…

TCP Splicing – CPU Usage

CPU Utilization under varying number of connections

TCP Splicing - Throughput

Throughput of a SOCKS and TCP Splices compared against IP Forwarding throughput

TCP Splicing - Latency

Distribution of Forwarding latency of TCP Splice, SOCKS and IP Forwarding

TCP Splicing - IssuesTCP Options

The client/proxy connection is negotiated before the proxy/server connection - it is possible that the 2 connections have negotiated incompatible options

1. Don’t splice incompatible connections

2. Proxy advertises minimal option set

3. Strip or map certain options as they pass through the TCP splice

See Maltz[3] for a fuller explanation on option mapping (MSS, SACK, etc.)

Host Mobility

On to host mobility…

Host Mobility - Intro

Host mobility is the idea that a computer can reattach itself to the network, without losing its connections.

Host Mobility – Mobile IPMobile IP uses the idea of a home agent, as a reflector to direct connections to the mobile host.Packets are sent using triangle routing.

Mobile host

Home Agent Foreign Client

Host Mobility – A New Take

Snoeren[2] introduces a transport layer solution to this problem, by introducing a new migrate-permitted TCP option.

Host Mobility – LocatingSnoeren’s Solution [2]:The first problem – How do you locate a machine’s IP address if it’s mobile?

No Problem – just use secure DNS updates to announce new location. (This is nothing new)

This is only an issue if the computer accepts passive connections.

Host Mobility - Migrating

Snoeren Introduces a new Migrate-Permitted TCP option

The mobile host will announce its new address to existing connections, and re-establish.

But how do we migrate existing connections?

Host Mobility – Migrating

In existing TCP/IP, connections are uniquely identified by<src addr, src port, dest addr, dest port>

With the Migrate TCP option, connections are identified by<src addr, src port, token>

Host Mobility – In Action

Mobile Host

Client TCP Connect


Mobile Host

Client TCP SessionEstablished


Mobile Host

Client

Mobile Host

Migrating…


Client

Mobile Host

TCP Connect& Migrate


Client

Mobile Host

TCP ConnectionRe-established

Host Mobility - Proxy

What if we want to use a proxy?

Well, that’s alright. In this way, only the TCP stacks of the proxy, and the mobile host need to be modified to deploy this scheme.

Of course, we’re back to the triangle routing/home agent approach…

Host Mobility – IssuesThis is an end to end solution. So, both client and server must support the migrate TCP. There needs to be financial incentive for wide-scale deployment.

Simultaneous migrations will not work.

Existing applications have often made assumptions about the stability of their network addresses.

Host MobilityNow that we’ve covered all these cool ideas, what can we do with them all?

On to the title paper –

Snoeren, Andersen, Balakrishnan,“Fine-Grained Failover Using ConnectionMigration”,Proc. of the Third Annual USENIX Symposium on Internet Technologies and Systems (USITS), March 2001

The Problem

This slide shamelessly copied from - http://nms.lcs.mit.edu/talks/usits01-migrate/sld002.htm

Servers Fail.More often than users want to know…

Solution: Server Redundancy

Use a healthy one at all times.

This slide shamelessly copied from - http://nms.lcs.mit.edu/talks/usits01-migrate/sld003.htm

Server Failover – In Action

Server A

Client TCP Connect

Server B

Health Monitor


Server A

ClientTCP SessionEstablished

Server B

Health Monitor


Server A

Client

Untimely Death..

Server B

Health Monitor


Client A

Client

Server B

Health Monitor

Death Notification


Client

Server B

TCP Connect& Migrate

Health Monitor

Server A


Client

Server B

TCP ConnectionRe-established

Health Monitor

Server A

So what Can we do?

So, now that we have this idea in mind, how can we use integrate this failover mechanism with existing services? (Incidentally, this architecture can also be used for load balancing)

We need a way to synchronize state across many machines. This idea works best if there is not much data to synchronize, and if our stream maps directly into a file.

A great example: static content on a web server farm

Web Farm Design

We want to keep compatibility with existing web servers

Let’s use a stream mapper to control all this fail-over mechanism. (aka. The Wedge)

Stream Mapper Stream Mapper

Server A Server B

Client

The WedgeLet’s take a closer look at the wedge

What great place to use a TCP Splice!

Synchronizing the Wedges

We need enough information to restart the TCP connection

Client addr/port

Takeover sequence #

TCP Migrate Fields (connection token)

Application specific object parameters (The URL)

What data do we need to synchronize between the wedges?

But we don’t need to sync last-sent packets!

Syncing the Streams

Why don’t we need to sync last-sent packets?

The client already has that information. All we need to do is elicit an ACK from the client, and they will tell us what the last sent packet is.

Let’s see how this happens -

Network TraceInitial Data Transmission: 0.00000 cl.1065 > sA.8080: . ack 0505 win 31856 --(Erroneous) sA Death Pronouncement Issued-- 0.08014 sA.8080 > cl.1065: P 0505:1953(1448) ack 1 win 31856Successful Connection Migration to sB: 0.09515 sB.1033 > cl.1065: S 0:0(0) win 0 <migrate PRELOAD1> 0.09583 cl.1065 > sB.1033: S 0:0(0) ack 1953 win 32120 0.14244 sB.1033 > cl.1065: . ack 1 win 32120Continued Data Transmission from sA: 0.17370 sA.8080 > cl.1065: P 0505:1953(1448) ack 1 win 31856 0.17376 cl.1065 > sA.8080: R 1:1(0) win 0Failed Connection Migration Attempt by sC: 0.17423 sC.1499 > cl.1065: S 0:0(0) win 0 <migrate PRELOAD1> 0.17450 cl.1065 > sC.1499: R 0:0(0) ack 1 win 0Resumed Data Transmission from sB: 0.24073 sB.1033 > cl.1065: P 1953:3413(1460) ack 1 win 32120 0.25663 cl.1065 > sB.1033: . ack 3413 win 31856 0.33430 sB.1033 > cl.1065: P 3413:4873(1460) ack 1 win 32120 0.42776 sB.1033 > cl.1065: P 4873:6333(1460) ack 1 win 32120 0.42784 cl.1065 > sB.1033: . ack 6333 win 31856 . . .

An annotated failover trace (collected at the client) depicting the migration of a connection to one of two candidate servers.

Wedge Overhead

The Request overhead of the wedge as a function of request size.

IssuesDual simultaneous migration is still not possible

Wedge definitely adds some overhead

Application level-sync is still non-trivial

- What if content changes?

- Still doesn’t solve interactive streams

- Some applications will require modifications to take advantage of mobility.

Some apps still assume static IP addresses

What if a server dies before announce its connections?

Needs wide-scale deployment

TCP Mobility/Splicing

Documents

Transcript of TCP Mobility/Splicing