CA*net 4 A Network for Grids using Grid

Technology

Bill St. Arnaud

CANARIE Inc – www.canarie.ca

Bill.st.arnaud@canarie.ca

CA*net 4 Drivers-1

> Set up lightpaths to no cost peering exchanges– Most lambda sales in Canada and USA are for “Remote peering” to

no cost peering points– Allows for considerable savings in Internet transit costs– Each lightpath is directly connected to a high volume peer and

bypasses peering router– Good example is “STAR LIGHT” where high volume peers have

direct connect and small volume peers use a router– CA*net 4 “customer controlled patch panel” allows peers to change

peering relationship remotely without contacting technical staff at peering exchange• Very similar in concept to WorldCom “Peermaker” at MAE-E and MAE-

W

CA*net 4 Drivers-2

> Eliminate expensive high end routers and replace them with lower cost optical switches– But circuits are NOT intended to replace packet networks– Use rich mesh of circuits between edge routers to eliminate high cost of

10GbE core routers– 10Gbe routers ~ $500K with interfaces at ~$200k each– 10Gbe switches ~$25K with interfaces at ~$20k each– Trade off between cost of multiple lightpaths versus cost of high end core

routers– 10Gbe wavelengths ~$1000/km for 5 years (lifetime of router)– Assume 1 GbE lightpaths per edge institution then

• One 5000km Gbe lightpath (or 8 x 600km GbE) lightpaths per institution is cheaper than routers

• But hard to create a full mesh – so let institutions or end user create and control partial mesh

– Disadvantage is no sharing of bandwidth with stat muxing

CA*net 4 Drivers-3

> Allows customer to create “customer owned and managed” networks with resource heterogeneity– Integration of wavelengths and dark fiber from different carriers– Create customer controlled VPNs for downstream users and

overlay networks across multiple suppliers– Customers can manage their own restoral and protection schemes– Allows for inter-domain end to end setup of VPNs– End users do not need to to signal carrier for VPN management

• Create VPNs

• Cross connect VPNs from independent users

• Partition or spawn VPNs

• Establish VPNs across multiple management domains

CA*net 4 Drivers-4

> Lambda Grids - “Underlay” networks to support Grids and overlay projects like PlanetLab and Oceanstore– A lot of exciting research into overlay networks – At some point in time when traffic volume is sufficient in overlay network

to setup its own direct path> Soon high end grid applications will have sufficient traffic volume to

require their own underlay networks ”Complementing” routed networks – Not a replacement for routed networks – only increasing the direct

peering mesh of the routed network– But peering may be more dynamic (and not globally advertised) than

traditional IP BGP peering> Discipline or applications specific networks

– VBLI grids like European EVN – High energy physics grid – Ultralight– NEES grid, Bio-informatics Grid, etc

Example – EVN traffic flows over GEANT

UK

SE

FR

NLBE

DE1DE2

CZ

PL

CH

IT

AT

SURFnet

JANET

GARR

PSNC

DFN

NORDUnet

2.5G10G

256M 512M ?1G

JIVE

Provided courtesy of Dai Davies

Issues

> How do you charge for bandwidth and usage when single application traffic dwarfs all other IP traffic?

> Who pays for the traffic volume when it sinks into POP?

> Possible solutions:– GMPLS (with QoS)

• Requires expensive routers and complex coordinated central management to setup and tear down tunnels

• Does not address issue of traffic charging

• Interdomain still unproven

– Optical overlay/underlay –ASON – same problems as GMPLS– Application specific optical BGP networks

CA*net 4 Drivers-5

> Spatial QoS – TCP throughput over long fat pipes very susceptible to packet

loss, MTU, TCP kernel, Buffer memory, AQM optimized for commodity Internet, Auto negotiating Ethernet, etc

– May also require consistent and similar TCP throughput for multiple sites to maintain coherency for grids and SANs

– Some exciting new TCP protocols like FAST, XCP, etc• Mice and Elephant problem• Without careful design may look like a DOS attack on a router

network

– Many commercial SAN/Grid products will only work with QoS network

– Some users want to have super jumbo MTU (64K) or protocols other than IP

Spatial QoS

Application or end user controls peering of BGP optical paths for transfer of elephants!!!

Normal BGP pathx.x.x.1 y.y.y.1

OBGP pathOnly y.y.y.1 advertised to x.x.x.1 via OBGP path

Only x.x.x.1 advertised to y.y.y.1 via OBGP path

Optical “Peermaker”

OBGP applied to EVN

UK

SE

FR

NLBE

DE1DE2

CZ

PL

CH

IT

AT

SURFnet

JANET

GARR

PSNC

DFN

NORDUnet

JIVE

CA*net 4 Drivers 6

> Extend the Internet end to end principle to circuit based networks– The success of the Internet is largely attributable to the e2e principle– No state maintained in the network– Allowed development of exciting new applications or services

> Can the same principles be applied to circuit based networks?– Will it engender the same creativity in new applications and

services?

> MPLS and ASON are classic network state based solutions for VPNs– CA*net 4 architecture is an alternate approach– All VPNs are BGP direct static routes using lightpaths

CA*net 4 is NOT a network

> It is an aggregation of point to point 10 Gbps wavelengths from a number of carriers

> CA*net 4 is made up of may parallel networks> The wavelengths and switches are partitioned into smaller

lightpaths user control of the switch partition which are used for a variety of applications particularly grids– International Grid Testbed – 10 Gbe server to server to CERN– WESTgrid – 1 Gbe lightpaths for distributed backplane– CA*net 4 IP network – traditional IP hierarchical routed network– Numerous lightpaths to support direct peering between regional networks

and universities– Lightpaths to support Terabyte file transfer from CERN for high energy

physics that bypasses all routers– Lightpaths to support TransLight projects between North America, Europe

and Asia– Many, many more coming – Virtual Astronomy, HDTV video walls, etc

RMI

Resource Management Layer

User Access Layer

Grid Application

Create service Access service

Web Server

DB

GT3 Hosting Environment

LPO FactoryService

Grid Service Interface

LPO Delegate Service J2EE Application Server

LPO Service

EJB Remote

LPO Service

EJB Home

JDBC

LPO Service

Implemen-tation

RMI

RMI

LPO Grid Service

Service Provisioning Layer

OGSI-conformant services LPO advertisement LPO query LPO termination LPO access LPO reconfiguration LPO spawning LPO concatenation End-to-end LPO establishment

OGSI-conformant services

Resource Agent

RMIService Provisioning Layer

LPO ControllerLPO Controller

RMI

TL1

Request Controller

Switch Interface