The application layer

Olof Hagsand, NADA/KTHolofh@nada.kth.se

January 20, 2005

0-0

The application layer

The TCP/IP application layer contains protocols that enable

applications to communicate.

The TCP/IP application layer roughly maps to three OSI layers:

• Session: session establishment, dialog control, synchronization

• Presentation: syntax and semantics of data: higher level data

structures

• Application: application-specific information and protocols

From its UNIX implementation roots, the definition of the the

application layer is: everything that is implemented in user space!

(not in the UNIX kernel).

1

Clients, servers, peers

Computers connected to the Internet are end-systems or hosts

(they “host” application programs running on them). Hosts are

traditionally divided into clients and servers - the difference

nowadays unclear.

But from a program point of view, it is easier:

• Client program - requests a service.

• Server program - provides a service.

• Peer - bot a client and a server program.

2

The Socket Interface

The socket interface is used for programming applications with a

network component.

Sometimes called BSD sockets - it was first implemented in C in

BSD.

Variants exist for most programming languages.

Winsock is almost the same but not quite!

Other programming interfaces include:

• Streams

• Remote Procedure Calls (RPC)

The sockets API is a de facto standard for network programming.

3

Protocol message formats

When you transfer information from one host to another, they need

to to understand each other’s data. (Presentation layer)

Protocol messages are designed in different ways, some issues:

• Performance - compact data for faster transmittal, easy to

parse by a computer.

• Readability - Easy to read by humans: debugging, surveillance,

editing.

• Common character sets - different languages, coding.

• Alignment and byte ordering - Different CPU characteristics.

4

Approach 1: Binary fixed fields

Most common in the underlying layers of the TCP/IP stack.

Examples: DNS, RIP, OSPFv2, BGP, RTP

Predefines exactly what information is to be where in the message.

The semantics is hard-coded into the application.

And its binary

5

Binary fixed fields (cont)

Requires common alignment (ie on 16, 32 or 64 -bit boundaries)

Requires byte-swapping: How the CPU loads its registers from

memory.

Two variants:

• Little endian (eg Intel): LSB (Least Significant Byte) first

• Big endian (eg Motorola): MSB (Most Significant Byte) first

Network byte order is big endian → You need to byte-swap on i386

PCs.

6

Example: DNS

The DNS header, taken from RFC 1035.

1 1 1 1 1 1

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5

+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

| ID |

+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

|QR| Opcode |AA|TC|RD|RA| Z | RCODE |

+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

| QDCOUNT |

+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

| ANCOUNT |

+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

| NSCOUNT |

+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

| ARCOUNT |

+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

7

Pros & Cons

When you feel the urge to design a [...] complex binary

application protocol, it is generally wise to lie down until

the feeling passes

Eric Raymond: ‘‘The Art of UNIX Programming’’

Pros:

• Compact: Efficient computer processing

• Fixed syntax and simple semantics

Cons:

• Not extendable

• Not human readable.

• Byte order, alignment problems

8

Approach 2: Tree based

Data structured hierarchical - recursive structure.

Both binary and textual variants.

More or less formal specification defining the data-types E.g. XML

DTD.

Examples: TLV, ASN.1, XML.

9

TLV - (Type, Length, Value)

Binary format usually used as an extensible part of a protocol.

• Type: contains a predefined code, indicating what kind of data

the value field contains.

• Length: Contains the size (in bytes) of the value field.

• Value: Contains the payload.

Examples: IS-IS and OSPFv3, DHCP, and IP options.

TLVs can be recursive (value field contains new TLVs).

But there is no notion of specification - must be added externally.

10

Example: DHCP

A vendor extension field taken from RFC 2132.

3.5. Router Option

The router option specifies a list of IP addresses for routers on the

client’s subnet. Routers SHOULD be listed in order of preference.

The code for the router option is 3. The minimum length for the

router option is 4 octets, and the length MUST always be a multiple

of 4.

Code Len Address 1 Address 2

+-----+-----+-----+-----+-----+-----+-----+-----+--

| 3 | n | a1 | a2 | a3 | a4 | a1 | a2 | ...

+-----+-----+-----+-----+-----+-----+-----+-----+--

11

Abstract Syntax Notation # 1

A general way to define data types.

ASN.1 is as powerful as a typed programming language.

In ASN.1 the type information is inherent in the data - no external

specification necessary.

Used frequently in ISO protocols, but also to a certain extent in

TCP/IP protocols.

Some examples: SNMP, UMTS, LDAP, NFSv4 and many security

protocols.

12

A tiny part of an SNMP definition:

PDU ::=

SEQUENCE {

request-id Integer32,

error-status INTEGER (

noError(0),

tooBig(1),

noSuchName(2),

badValue(3),

readOnly(4),

...

inconsistentName(18)

),

error-index INTEGER (0..max-bindings),

variable-bindings VarBindList

}

13

XML

Plain-text markup language: simple syntax, easy to parse.

Definition declared externally by XML Schema or DTD.

Well suited for complex data formats with recursive and nested

structures.

Cons mainly its textual nature: parsing can be inefficient.

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE note SYSTEM "InternalNote.dtd">

<note>

<to>Eva</to>

<from>Phil</from>

<heading>Reminder</heading>

<body>Remember to go to the store!</body>

</note>

14

Approach 3: RFC 822 formats

Classical Internet format described by BNF (Backus-Naur Form) -

derived from context-free grammars.

Several RFCs describes the actual syntax description: RFC 822,

RFC 2068, RFC 2234, now called ABNF - Augmented BNF.

RFC 822 is syntax-heavy: keywords are introduced for parsing,

requires specific parsers.

15

RFC 822 based text protocols (con’t)

For example:

name = elements crlf a rule

crlf = %d13.d10 characters to end a line

‘‘literal’’ a string, case insensitive

element1 / element2 an alternative

(element1 element2) a strict sequence

DIGIT = %x30-39 a range of characters

<a>*<b>element element repetition

[foo bar] optional elements

. . . and more . . .

16

RFC 822 based text protocols (con’t)

Another example; In RFC 2048, the HTTP URL is defined as:

http URL = ‘‘http:’’ ‘‘//’’ host [ ‘‘:’’ port ] [abs path ]

host = A legal Internet host domain name or IP

address (in dotted-decimal form) as defined by

Section 2.1 of RFC 1123

port = *DIGIT

abs path = ‘‘/’’ rel path

rel path = [ path ] [ ‘‘;’’ params ] [ ‘‘?’’ query ]

17

Pros & Cons

• Pros:

– Easy to extend and flexible.

– Human readable (easy to debug)

• Cons:

– Not compact.

– Syntax-heavy: may require complex parsers.

18

Specific applications/protocols

• telnet

• http

• tftp

• ftp

• smtp

• snmp

• rtp

• sip

• Others: Instant Messaging, Peer-to-peer, Distributed gaming.

19

TELNET - Terminal Network

(TCP port 23, text) Virtual Terminal local terminal appears to be

a terminal on a remote system

It is a nice tool to test other text-based protocols (HTTP, SMTP,

FTP, etc)

Good example of interactive application

• Tinygrams leading to silly window syndrome:

• Nagle’s algorithm

• Delayed ack, etc

Control: simple options (control bytes have first bit set)

TELNET is security challenged: use TELNET with Kerberos or

SSH!

20

HTTP

(TCP port 80, RFC 2616, ABNF data)

The Hypertext Transfer Protocol is the main protocol used to

download resources from the world wide web.

Simplest form: a requestor establishes a TCP connection to the

web server on port 80 and sends a string describing what resource

it wants, and receives the resource in reply.

The most modern version today is HTTP/1.1.

21

HTTP - example

-> GET /stuff/blah.html HTTP/1.1

-> Host: zipf.pilsnet.sunet.se

-> User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031214 Firebird/0.7

-> Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1

-> Accept-Language: en-us,en;q=0.5

-> Accept-Encoding: gzip,deflate

-> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

-> Keep-Alive: 300

-> Connection: keep-alive

<- HTTP/1.1 200 OK

<- Date: Tue, 27 Jan 2004 20:18:28 GMT

<- Server: Apache/1.3.27 (Unix) (Gentoo/Linux) PHP/4.3.4

<- Last-Modified: Tue, 27 Jan 2004 19:53:47 GMT

<- ETag: "bb4047-2c-4016c1cb"

<- Accept-Ranges: bytes

<- Content-Length: 44

<- Keep-Alive: timeout=15, max=100

<- Connection: Keep-Alive

<- Content-Type: text/html

<-

<- <html>

<- <b>

<- Hello there

<- </b>

<- </html>

22

Some HTTP commands

• GET http url: Download an http resource.

• POST http url: Upload data to an http resource.

• PUT http url: Write an http resource.

• DELETE http url: Delete an http resource.

23

Some HTTP status codes (con’t)

Some examples:

• 200 Ok

• 404 Not found

• 301 Moved Permanently

• 500 Internal Server Error

24

HTTP 1/1 persistent connections

In HTTP 1/0, all HTTP requests generated a new TCP connection.

But most html documents contain sub-parts → one TCP

connection for each sub-request.

But TCP congestion control is made for longer connections → they

can adapt to congestion in the network.

When http traffic grew when the web exploded, these small flows

were said to kill the Internet!

HTTP 1/1 supports persistent connections: keep the TCP

connection during the complete session: send all requests on the

same TCP connection.

Now, these longer TCP connection can perform congestion control

algorithm in a proper way.

25

TFTP - Trivial File Transfer Protocol

(Text-based, UDP port 69, RFC 1350)

Very simple protocol to transfer files.

Character coding: netascii(like telnet) or binary.

Stop-and-go protocol: send datagram, wait for ack.

Small implementations: typically on boot PROMS for small devices

and diskless clients.

Five message types:

• RRQ - Read ReQuest

• WRQ - Write ReQuest

• DATA

• ACK

• ERROR

26

FTP - File Transfer Protocol

(Text-based, TCP ports 20 and 21, RFC 959)

FTP is a more elaborate file transfer protocol.

FTP conducts its sessions in clear text.

FTP uses two TCP connections:

• The control connection - exchange commands and their replies.

TCP session initiated by the client to the server on port 21.

• The data connection - to transfer data in a specified mode and

type. Data transferred may be a part of a file, an entire file or

a number of files.

27

FTP modes

FTP can run in two modes- active mode and passive mode. This

refers to whether the ftp server will start the data connection or

not.

• active: The server will start the TCP session for the data

connection, thereby connecting to the client to a port and IP

specified by the client. (May not work if client is behind NAT)

• passive: The server will not start a TCP session. Instead, the

client will create a TCP session to the server, to a port and IP

specified by the server.

28

Some FTP commands

Examples of FTP control commands (sent on control channel):

• CWD <arg> Change working directory

• RMD <arg> Remove directory

• PWD Print working directory

• TYPE [I|A|E|L <arg>] Set the data transfer type

• RETR <arg> Download a file.

• STOR <arg> Upload a file.

• LIST Download the current working directory’s content list.

29

Some FTP status codes

As in HTTP, FTP has a variety of status codes:

• 1xx Positive Preliminary reply

The requested action is being initiated; expect another reply before proceeding

with a new command.

• 2xx Positive Completion reply

The requested action has been successfully completed. A new request may be

initiated.

• 3xx Positive Intermediate reply

The command has been accepted but the requested action is waiting for further

information before being completed.

• 4xx Transient Negative Completion reply

The command was not accepted and the requested action did not take place,

but the error condition is temporary and the action may be requested again.

• 5xx Permanent Negative Completion reply

The command was not accepted and the requested action did not take place.

30

SMTP - Simple Mail Transfer Protocol

(Text-based, TCP port 25, RFC 2821)

SMTP the protocol to transfer email from hosts to mail servers and

between mail servers.

Terminology:

• User Agent(UA) - end-hosts.

• Mail Transfer Agent (MTA) - mail servers.

Addressing: <mailbox>@<domain name>

This results in a a DNS MX request for <domain name>, giving a

name of the MTA to transfer the message to.

31

SMTP (cont)

Electronic mail is different from the previous protocols in its

delayed delivery in several steps:

• Spooling from sending host to first MTA.

• Relaying by intermediate MTAs.

• Downloading of email by receiving host using other protocols:

POPv3 (Post Office Protocol) or IMAPv3 (Interactive Mail

Access Protocol)

32

SMTP syntax

Like HTTP and FTP, SMTP has special commands and status

codes.

• HELO <hostname>:

• MAIL FROM <email address>: Sender email address

• RCPT TO <email address>: Recipient email address

• DATA: Tells the email server that data follows.

• QUIT: Immediately close the connection.

The status codes are similar to that of HTTP and FTP.

33

MIME - Multipurpose Internet Mail Extensions

Classical email messages must be written in US-ASCII (7-bit).

What does this imply?

MIME aims at redefining the format of messages to allow for:

• textual message bodies in character sets other than US-ASCII,

• an extensible set of different formats for non-textual message

bodies,

• multi-part message bodies, and

• textual header information in character sets other than

US-ASCII

34

So how does it work?

Related header fields:

• Content-Type - what kind of data the content carries.

Some examples: text/plain, text/html, audio, video,

application/pdf, extension-token, and multipart.

• Content-Transfer-Encoding - how data is encoded.

Some examples: 7bit, 8bit, binary, quoted-printable,

base64,...

35

SNMP Simple Network Management Protocol

(TCP, ASN.1)

• It is complex to build internetworks and we need to manage

them.

– Monitoring

– Debugging

– Control routers and other network devices

• SNMP - Internet management

– No special control messages use TCP/IP itself

– Management is on TCP/IP application level

– Same protocol is used for all managed devices

– If IP does not work correctly,...

36

Real-time multimedia

Time-sensitive, interactive applications: (eg, telephony).

Use RTP- Real-Time Protocol.

Limited time-sensitivity: Streaming protocols.

Use RSTP (Real-Time Streaming Protocol)

Non-time sensitive: Transfer the data using file transfer.

37

Signaling

So, RTP can be used to transfer time-sensitive data streams.

But what about signaling: how to set up sessions:

• SIP - Session Initization Protocol

• H.323

38

SIP

(TCP or UDP port 5060, ABNF)

Terminology is similar to SMTP, but is a synchronous protocol (no

delays).

SIP uses URI’s (Uniform Resource Identifiers) as addresses:

<sip:6534@kth.se>

<sip:bob@biloxi.com>

SIP uses transactions, usually three-way (as TCP connections).

Example:

INVITE → 200 OK → ACK

39

SIP Example

From RFC 3261:

softphone proxy proxy SIP Phone

| | | |

| INVITE F1 | | |

|--------------->| INVITE F2 | |

| 100 Trying F3 |--------------->| INVITE F4 |

|<---------------| 100 Trying F5 |--------------->|

| |<-------------- | 180 Ringing F6 |

| | 180 Ringing F7 |<---------------|

| 180 Ringing F8 |<---------------| 200 OK F9 |

|<---------------| 200 OK F10 |<---------------|

| 200 OK F11 |<---------------| |

|<---------------| | |

| ACK F12 |

|------------------------------------------------->|

| Media Session |

|<================================================>|

| BYE F13 |

|<-------------------------------------------------|

| 200 OK F14 |

|------------------------------------------------->|

40

SIP message example

INVITE sip:000730631661@kth.se SIP/2.0

Via: SIP/2.0/UDP 192.36.125.167:5060;branch=z9hG4bK0e4415ea

From: "6534" <sip:6534@kth.se>;tag=000e38a3b7e8001d597d1d53-1bfa7620

To: <sip:000730631661@kth.se>

Call-ID: 000e38a3-b7e8001e-34c94c48-72c83866@192.36.125.167

Date: Mon, 03 Jan 2005 14:16:06 GMT

CSeq: 101 INVITE

User-Agent: CSCO/6

Contact: <sip:6534@192.36.125.167:5060>

Expires: 180

Content-Type: application/sdp

Content-Length: 251

Accept: application/sdp

41

IM - Instant Messaging

On-line messaging and presence information using a central server

and many connected clients.

Some systems: AOL IM/ICQ, MSN Messenger, Yahoo Messenger

An IM system typically has the following features:

• Buddy list

• Chat, Images, Sounds, File-sharing

• Real-time talk and video

Most protocols are proprietary. But SIP has messaging extensions

(SIMPLE).

A special feature is to serve many small messages in a short time,

and to manage presence information.

42

Peer-to-peer file-sharing applications

Example of content-distribition (file-sharing) using peer-to-peer

techniques.

Build overlays – virtual networks on top of physical network.

Overlay links are TCP/UDP connections.

Usually, actual data transfer is direct between hosts (peer-to-peer),

often using HTTP.

Some have central registry (index of where files are) (Napster).

Others (eg KaZaa) have distributed registry: some nodes with good

network connections, no NAT, and large resources turn into

supernodes. All clients connect to a supernode.

43

Peer-to-peer file-sharing applications (cont)

Some are completely decentralized (GnuTella), encrypts data

(FreeNet).

BitTorrent, for example, works closely with HTTP - splitting up an

HTTP transfer in slices, distributing the download from one

originator to many clients working in unison.

Many rely on distributed hash lookup functions to make fast

queries and lookup of data.

Some of the routing problems are similar to real (physical) routing,

but on a higher level.

44

Detour: NAT traversal

Nowadays, most hosts are behind NAT (Network Address

Translation) boxes

NATs translate global IP addresses to local, and extends the

address space using TCP/UDP ports.

One peer behind NAT: possible to initiate connection from behind

a NAT.

Both peers behind NATs: difficult to communicate directly.

Solution: For UDP, exploit some regularities of NATs (reuse of

same ports, etc). Or use a non-NAT peer as “protocol bouncer”.

45

Skype

(Encrypted, TCP/UDP)

Skype is a VoIP tool using peer-to-peer techniques for name-lookup.

Skype is a completely closed system - no open interfaces, not even

which RFCs are implemented → No interoperation possible.

You could say this violates the Internet spirit.

• Uses high compression: iLBC coding (≈ 10x compression of

audio data)

• Name lookup using same infrastructure as KaZaa: nodes and

supernodes.

• NAT traversal techniques using UDP, TCP or “bounce”

connections via supernodes.

• End-to-end RSA encryption

46

Distributed games

Some of the best-known distributed games are interactive and

real-time: Doom, Quake, Counter-Strike, Half-life, etc.

Some issues are:

• Low latency: “low pingers” win fights. Usually small UDP

packets.

• Textures and geometric information preloaded: only deltas

distributed.

• Movement of 3D graphics may use “dead reckoning”: no need

to send updates on all geometric movements: use motion

equations instead.

• All communication via central server, synchronizes and resolves

events (who wins a fight).

47