MB - NG MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester MB – NG SuperJANET4 Development Network...

MB-NG Meeting Dec 2001R. Hughes-Jones Manchester

MB - NG

MB – NG SuperJANET4 Development Network

SuperJANET4ProductionNetwork

Leeds

RAL /UKERNA

RAL /UKERNA

SJ4 DevC-PoP

Reading12416

SJ4 DevC-PoP

Warrington12416

Manc MCC

ULCC UCL

SJ4 DevC-PoPLondon12416

Gigabit Ethernet2.5 Gbit POS2.5 Gbit POS core

WorldCom

WorldCom

Dark Fibre (SSE)


MB - NG Some Edge (Bbone) Router requirements

Connect to the test systems in the IP domain. Accept marked packets and be able to re-mark them Mark the packets on input according to at least:

Source – dest IP address IP portLayer3 and Layer4 headers TCP/UDPSupport Diffserve code points

Perform policing and admission control Provide a range of Queue type options –

Several queues with different prioritiesReal-time queue

Provide flexible Queue SchedulingWeighted round robinWeighted fair queuing

Provide Congestion control such as RED ECN

Be able to assign MPLS labels to the packets in a flexible manner.


MB - NG

Equipment for the External / Edge DomainsFunctionality

Load PC

Test PC

Test PC

Load PC

Test PC

Load PC

Test PC

Load PC

Difserve routerORMPLS Edge Router

4 * 1 Gigabit Ethernet bladeQoS Marking

Gigabit Ethernet2.5 Gbit POS2.5 Gbit POS core

SJDN CoreMPLS Backbone Router 12416

3 * 1 Gigabit Ethernet blade

2.5 Gbit POS

10 Gbit POS

10 Gbit POS


MPLS

2.5 Gbit POSMPLS

MPLS Backbone Router OSR

IP orMPLS

Test PC

Supervisor Supervisor

Load PC

MPLS

Classify packetsQoS Difserv

QoS – WREDMPLS

Generate MPLS frames

MPLSMulti-loop


MB - NG

Ethernet Switch 3512

Local site Ethernet

Ethernet Switch 3512

Local site Ethernet

Equipment for the External / Edge DomainsConnectivity

Load PC

Test PC

Test PC

Load PC

Test PC

Load PC

Difserve routerORMPLS Edge Router

4 * 1 Gigabit Ethernet bladeQoS Marking

Gigabit Ethernet2.5 Gbit POS2.5 GbitPOS core


MPLS

2.5 Gbit POSMPLS

MPLS Backbone Router OSR

Spy

Test PC

Supervisor Supervisor

Load PC

MPLS to core

3508

Monitor PC


MB - NG IP Access to MB-NG Systems

A suggestion only – we all must decide !

End/Test systems routing: The default route from the PC is set to the Gigabit link

i.e. into the development network. Static routes for researchers only are set for the 100

Mbit interface, e.g. to Manchester HEP, MCC, RAL, UCL, ULCC, UKERNA.

The link between the Cisco 3512 and the Campus is set to 10 Mbit – this would minimise any unintentional traffic.

IP access to core routers for researchers only Similar approach? via the test PCs at the CPoP


MB - NG Some Questions ???

Can one Gigabit Ethernet blade perform the IP and MPLS functions we need or do we require two cards – one for IP input functions, the other for MPLS output?

What is the performance of the 4-port Gigabit Ethernet interfaces?

Confirm that an interface can support both IP and MPLS frames at same time.

Can the GSRs use MPLS to loop the frames round n times?

Can all our “requirements be” met?


MB - NG

Cisco OSR based on 6500 Are the details correct ?

Catalyst 6509 Chassis w/ 2500W AC Power SupplyCatalyst 6000 2500W AC Power SupplyCatalyst 6500 Supervisor Engine-2, 2GE, plus MSFC-2 & PFC-2Catalyst 6000 MSFC2 IOS ENTERPRISE/FW W/VIPCatalyst 6500 Switch Fabric ModuleCatalyst 6000 Supervisor PCMCIA Flash Mem Card, 24MB Option

4-port Gigabit Ethernet Optical Services Module, GBIC1-port OC-48/STM-16 SONET/SDH OSM, SM-SR, with 4 GE128 MB ECC Memory for Optical Services Modules1000BASE-SX "Short Wavelength" GBIC (Multimode only)1000BASE-LX/LH "long haul" GBIC (singlemode or multimode)1000Base-ZX extended reach GBIC(singlemode)

Focus on the routers - Hardware

More memory


MB - NG

Focus on the routers - Summary

C-PoP GSR kit Backbone OSR kit

Edge OSR kit

MCC Chassis Supv etc 1 OSR Ethernet 1 OSR 2.5G POS 1 GSR POS 2 3512 Eht. switch 1 1 M/c HEP Chassis Supv etc 1 OSR Ethernet 1 OSR 2.5G POS 1 3512 Eht. switch 1 3508 Eht. switch 1 RAL1 Chassis Supv etc 1 OSR Ethernet 1 GSR 3 GigEth 1 OSR 2.5G POS 3512 Eht. switch 1 1 RAL2 Chassis Supv etc 1 OSR Ethernet 1 OSR 2.5G POS 3512 Eht. switch 1 3508 Eht. switch 1 ULCC/UCL1 Chassis Supv etc 1 OSR Ethernet 1 OSR 2.5G POS 1 GSR POS 1 3512 Eht. switch 1 1 UCL2 Chassis Supv etc 1 OSR Ethernet 1 OSR 2.5G POS 3512 Eht. switch 1 3508 Eht. switch 1


MB - NG

Hardware & Software Tools

•Optical splitter or opto-electrical tap•opto-electrical tap constructed for logic analyser

Software use of :tcpdump tcptraceWeb100


Test PC

2.5 Gbit POS

Monitor PC

NIC

NIC

High Bandwidth High Throughput Protocol Investigations (mainly)


Test PC

2.5 Gbit POS

Monitor PCNIC

EthernetSwitch3508

Monitor port


MB - NG

Latency Tests with 64-bit PCI

Alteon Gigabit NIC + RAW SuperMicro 370DLE

ServerWorks III LE Chipset 64 bit PCI @ 33 MHz

Request – Response Transfer rates: observe expect us/B

32+32 bit 0.0265 0.0232

-> 108Mbyte/s

32+64 bit 0.0209 0.0211

64+64 bit 0.0149 0.0156

y = 0.0265x + 54.68

y = 0.0149x + 54.001

y = 0.0209x + 57.501

0

20

40

60

80

100

0 200 400 600 800 1000 1200 1400

Message length bytes

La

ten

cy

us

Alteon32 -> Alteon32



64 byte Request 1400 byte Response


MB - NG Throughput Tests with 64-bit PCI

RAW Stream Not using IP stack Vary time between

packets 1.6 factor

improvement over 32-32 bit transfers

Comparison 32 and 64 bit interfaces

0

200

400

600

800

1000

1200

0 200 400 600 800 1000 1200 1400 1600

Packet size Bytes

Re

cv

Wir

e r

ate

Mb

it/s

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Th

rou

gh

pu

t R

ati

o

32 bit64 bitRatio 64/32

Alteon 64bit - 64bit

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40 45

Transmit Time per frame usR

ecv W

ire r

ate

Mb

its/s

50100200400600800100012001400


MB - NG Throughput Tests UDP SysKonnect

UDP Stream Vary time between packets 660-690 Mbit/s @ 17us No interrupt coalescing

15 μs between packets

UDP SysKonnect 64 bit

0

100

200

300

400

500

600

700

800

0 5 10 15 20 25 30 35 40Transmit Time per frame us

Recv W

ire r

ate

Mb

its/s

50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes

1400 byte sent

1400 byte received

Pci-pci latency 28.3 μs


MB - NG Latency Tests UDP SysKonnect

UDP request - response Latency and transfer rates

very similar to raw frames

observe us/B offset us

64+64 bit 0.0225 46.88

64+64 bit 0.0149 54. (raw)

1400 byte sent10.66 μs pci time

1400 byte received10.96 μs pci time

Pci-pci latency 28.3 μs

y = 0.0252x + 46.887

y = 0.0127x + 64.183

0

20

40

60

80

100

0 500 1000 1500 2000 2500 3000


Late

ncy u

s

PC=PC UDP SysKonnect


MB - NG Throughput Tests UDP Alteon

UDP Stream Vary time between packets 880-930 Mbit/s @ 11us interrupt coalescing

15 μs between packets

UDP Alteon

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40

Transmit Time per frame us

Rec

v W

ire

rate

Mb

its/

s

50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes


MB - NG Latency Tests UDP Alteon

UDP request – response Due to interrupt coalescing

y = 0.0265x + 54.68

y = 0.0149x + 54.001

y = 0.0209x + 57.501

0

20

40

60

80

100

0 200 400 600 800 1000 1200 1400


La

ten

cy

us




0

50

100

150

200

250

300

350

400

0 500 1000 1500 2000 2500 3000

Message length bytesL

aten

cy u

s

PC=PC Alteon

MB - NG MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester MB – NG SuperJANET4 Development Network...

Documents

Transcript of MB - NG MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester MB – NG SuperJANET4 Development Network...