Connection-Based vs. Connectionless

• Telephone: operator sets up connection between the caller and the receiver– Once the connection is established, conversation can continue

for hours• Share transmission lines over long distances by using

switches to multiplex several conversations on the same lines– “Time division multiplexing” divide B/W transmission line into a

fixed number of slots, with each slot assigned to a conversation• Problem: lines busy based on number of conversations,

not amount of information sent• Advantage: reserved bandwidth

Connection-Based vs. Connectionless

• Connectionless: every package of information must have an address => packets – Each package is routed to its destination by looking

at its address– Analogy, the postal system (sending a letter)– also called “Statistical multiplexing”– Note: “Split phase buses” are sending packets

Routing Messages• Shared Media

– Broadcast to everyone• Switched Media needs real routing. Options:

– Source-based routing: message specifies path to the destination (changes of direction)

– Virtual Circuit: circuit established from source to destination, message picks the circuit to follow

– Destination-based routing: message specifies destination, switch must pick the path» deterministic: always follow same path» adaptive: pick different paths to avoid congestion, failures» Randomized routing: pick between several good paths to

balance network load

• mesh: dimension-order routing– (x1, y1) -> (x2, y2)– first x = x2 - x1,– then y = y2 - y1,

• hypercube: edge-cube routing– X = xox1x2 . . .xn -> Y = yoy1y2 . . .yn

– R = X xor Y– Traverse dimensions of differing

address in order• tree: common ancestor

Deterministic Routing Examples

001

000

101

100

010 110

111011

Store and Forward vs. Cut-Through• Store-and-forward policy: each switch waits for the full

packet to arrive in switch before sending to the next switch (good for WAN)

• Cut-through routing or worm hole routing: switch examines the header, decides where to send the message, and then starts forwarding it immediately – In worm hole routing, when head of message is blocked,

message stays strung out over the network, potentially blocking other messages

– Cut through routing lets the tail continue when head is blocked, accordioning the whole message into a single switch. (Requires a buffer large enough to hold the largest packet).

Store and Forward vs. Cut-Through• Advantage

– Latency reduces from function of:

number of intermediate switches X by the size of the packet

to time for 1st part of the packet to negotiate the switches + the packet size ÷ interconnect BW

Congestion Control• Packet switched networks do not reserve bandwidth; this

leads to contention (connection based limits input)• Solution: prevent packets from entering until contention is

reduced (e.g., freeway on-ramp metering lights)• Options:

– Packet discarding: If packet arrives at switch and no room in buffer, packet is discarded (e.g., UDP)

– Flow control: between pairs of receivers and senders; use feedback to tell sender when allowed to send next packet» Back-pressure: separate wires to tell to stop» Window: give original sender right to send N packets before getting

permission to send more; overlaps latency of interconnection with overhead to send & receive packet (e.g., TCP), adjustable window

– Choke packets: aka “rate-based”; Each packet received by busy switch in warning state sent back to the source via choke packet. Source reduces traffic to that destination by a fixed % (e.g., ATM)

Practical Issues for Inteconnection Networks

• Standardization advantages:– low cost (components used repeatedly)– stability (many suppliers to chose from)

• Standardization disadvantages:– Time for committees to agree– When to standardize?

» Before anything built? => Committee does design?» Too early suppresses innovation

• Perfect interconnect vs. Fault Tolerant?– Will SW crash on single node prevent communication?

(MPP typically assume perfect)• Reliability (vs. availability) of interconnect

Practical IssuesInterconnection MPP LAN WANExample CM-5 Ethernet ATMStandard No Yes YesFault Tolerance? No Yes YesHot Insert? No Yes Yes

• Standards: required for WAN, LAN!• Fault Tolerance: Can nodes fail and still deliver

messages to other nodes? required for WAN, LAN!• Hot Insert: If the interconnection can survive a failure,

can it also continue operation while a new node is added to the interconnection? required for WAN, LAN!

Cross-Cutting Issues for Networking

• Efficient Interface to Memory Hierarchy vs. to Network– SPEC ratings => fast to memory hierarchy– Writes go via write buffer, reads via L1 and L2

caches• Example: 40 MHz SPARCStation(SS)-2 vs 50

MHz SS-20, no L2$ vs 50 MHz SS-20 with L2$ I/O bus latency; different generations

• SS-2: combined memory, I/O bus => 200 ns• SS-20, no L2$: 2 busses +300ns => 500ns• SS-20, w L2$: cache miss+500ns => 1000ns

Protocols: HW/SW Interface

• Internetworking: allows computers on independent and incompatible networks to communicate reliably and efficiently;– Enabling technologies: SW standards that allow reliable

communications without reliable networks– Hierarchy of SW layers, giving each layer responsibility for portion

of overall communications task, called protocol families or protocol suites

• Transmission Control Protocol/Internet Protocol (TCP/IP)– This protocol family is the basis of the Internet– IP makes best effort to deliver; TCP guarantees delivery– TCP/IP used even when communicating locally: NFS uses IP even

though communicating across homogeneous LAN

Protocol

• Key to protocol families is that communication occurs logically at the same level of the protocol, called peer-to-peer, but is implemented via services at the lower level

• Danger is each level increases latency if implemented as hierarchy (e.g., multiple check sums)

TCP/IP packet

• Application sends message• TCP breaks into 64KB

segements, adds 20B header• IP adds 20B header, sends to

network• If Ethernet, broken into

1500B packets with headers, trailers

• Header, trailers have length field, destination, window number, version, ...

TCP data(Š 64KB)

TCP Header

IP Header

IP Data

Ethernet

Example Networks

• Ethernet: shared media 10 Mbit/s proposed in 1978, carrier sensing with expotential backoff on collision detection

• Multiple Ethernets with devices to allow Ethernets to operate in parallel!

• 10 Mbit Ethernet successors?– ATM (too late?)– Switched Ethernet– 100 Mbit Ethernet (Fast Ethernet)– Gigabit Ethernet

Connecting Networks• Bridges: connect LANs together, passing traffic from

one side to another depending on the addresses in the packet. – operate at the Ethernet protocol level– usually simpler and cheaper than routers

• Routers or Gateways: these devices connect LANs to WANs or WANs to WANs and resolve incompatible addressing. – Generally slower than bridges, they operate at the

internetworking protocol (IP) level – Routers divide the interconnect into separate smaller subnets,

which simplifies manageability and improves security• Cisco is major supplier;

basically special purpose computers

Example Networks

Length (meters)Number data linesClock RateSwitch?Nodes (N)MaterialBisection BW

(Mbit/s)Peak Link BW

(Mbits/s)Measured Link BW

IBM SP-2

10

8

40 MHz

Yes

Š512 copper

320xNodes

320

284

100 Mb Ethernet

200

1

100 MHz

No

Š254 copper

100

100

--

MPP LANATM

100/1000

1

155/622…

Yes

10000copper/fiber

155xNodes

155

80

WAN

Example Networks (cont’d)

Latency (µsecs) Send+Receive

Ovhd (µsecs)TopologyConnectionless?Store & Forward?Congestion

ControlStandardFault Tolerance

IBM SP-2

1

39

Fat tree

Yes

No

Back-pressure

No

Yes

100 Mb Ethernet

1.5

440

Line

Yes

No

Carrier Sense

Yes

Yes

ATM

50

630

Star

No

Yes

Choke packets

Yes

Yes

MPP LAN WAN

Examples: Interface to Processor

Packet Formats

• Fields: Destination, Checksum(C), Length(L), Type(T)• Data/Header Sizes in bytes: (4 to 20)/4, (0 to 1500)/26, 48/5

Example Switched LAN Performance

Network Interface Switch Link BWAMD Lance Ethernet Baynetworks 10 Mb/s

EtherCell 28115Fore SBA-200 ATM Fore ASX-200 155 Mb/s Myricom Myrinet Myricom Myrinet 640 Mb/s • On SPARCstation-20 running Solaris 2.4 OS• Myrinet is example of “System Area Network”: networks

for a single room or floor: 25m limit– shorter => wider faster, less need for optical– short distance => source-based routing => simpler switches– Compaq-Tandem/Microsoft also sponsoring SAN, called

“ServerNet”

Example Switched LAN Performance (1995)

Switch Switch LatencyBaynetworks 52.0 µsecs

EtherCell 28115Fore ASX-200 ATM 13.0 µsecsMyricom Myrinet 0.5 µsecs

– Measurements taken from “LogP Quantyified: The Case for Low-Overhead Local Area Networks”, K. Keeton, T. Anderson, D. Patterson, Hot Interconnects III, Stanford California, August 1995.

UDP/IP performance

Network UDP/IP roundtrip, N=8BFormulaBay. EtherCell 1009 µsecs +2.18*NFore ASX-200 ATM 1285 µsecs +0.32*NMyricom Myrinet 1443 µsecs +0.36*N• Formula from simple linear regression for tests from N

= 8B to N = 8192B• Software overhead not tuned for Fore, Myrinet;

EtherCell using standard driver for Ethernet

NFS performance

Network Avg. NFS response LinkBW/EtherUDP/E.

Bay. EtherCell 14.5 ms1 1.00Fore ASX-200 ATM 11.8 ms15 1.36Myricom Myrinet 13.3 ms 64 1.43• Last 2 columns show ratios of link bandwidth and

UDP roundtrip times for 8B message to Ethernet

Estimated Database performance (1995)

Network Avg. TPS LinkBW/E.TCP/E.Bay. EtherCell 77 tps1 1.00Fore ASX-200 ATM 67 tps15 1.47Myricom Myrinet 66 tps 64 1.46• Number of Transactions per Second (TPS) for

DebitCredit Benchmark; front end to server with entire database in main memory (256 MB)– Each transaction => 4 messages via TCP/IP– DebitCredit Message sizes < 200 bytes

• Last 2 columns show ratios of link bandwidth and TCP/IP roundtrip times for 8B message to Ethernet

Summary: Networking

• Protocols allow hetereogeneous networking– Protocols allow operation in the presense of

failures– Internetworking protocols used as LAN protocols

=> large overhead for LAN• Integrated circuit revolutionizing networks as

well as processors– Switch is a specialized computer– Faster networks and slow overheads violate of

Amdahl’s Law