1

ESnet Update Summer 2007

Joint Techs Workshop

Joe Burrescia ESnet General Manager

July 16,2007

Energy Sciences Network Lawrence Berkeley National Laboratory

Networking for the Future of Science

2

TWC

SNLL

YUCCA MT

PNNL LIGO

INEEL

LANL

SNLA Allied Signal

ARM

KCP

NOAA

OSTI ORAU

SRS

JLAB

PPPL

Lab DC Offices

MIT

ANL

BNL

FNAL AMES

NR

EL

LLNL

GA

DOE-ALB

OSC GTN NNSA

International (high speed) 10 Gb/s SDN core 10G/s IP core 2.5 Gb/s IP core MAN rings (≥ 10 G/s) Lab supplied links OC12 ATM (622 Mb/s) OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less

NNSA Sponsored (12) Joint Sponsored (3) Other Sponsored (NSF LIGO, NOAA) Laboratory Sponsored (6)

42 end user sites

SINet (Japan) Russia (BINP) CA*net4

France GLORIAD (Russia, China) Korea (Kreonet2

Japan (SINet) Australia (AARNet) Canada (CA*net4 Taiwan (TANet2) Singaren

ESnet IP core: Packet over SONET

Optical Ring and Hubs

ELP

commercial peering points

MAE-E

PAIX-PA Equinix, etc.

PNW

GPo

P/

PAci

ficW

ave

ESnet 3 with Sites and Peers (Early 2007)

ESnet core hubs IP

Abilene high-speed peering points with Internet2/Abilene

Abilene

CERN (USLHCnet

DOE+CERN funded)

GÉANT - France, Germany, Italy, UK, etc

SNV

JGI LBNL

SLAC NERSC

SNV SDN

SDSC

Equinix

SNV

ALB

ORNL

CHI

MREN Netherlands StarTap Taiwan (TANet2, ASCC)

NA

SA A

mes

AU

AU

SEA

CH

I-SL MA

N L

AN

A

bile

ne

Specific R&E network peers Other R&E peering points

AMPATH (S. America) AMPATH

(S. America)

ESne

t Sc

ienc

e Dat

a Net

work

(SD

N) co

re

R&E networks

Office Of Science Sponsored (22)

ATL

NSF/IRNC funded

Equinix

3 ESnet 3 Backbone as of January 1, 2007

Sunnyvale

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) MAN rings (≥ 10 G/s) Lab supplied links

Seattle

San Diego Albuquerque

El Paso

Chicago New York

City

Washington DC

Atlanta

Future ESnet Hub

ESnet Hub

4 ESnet 4 Backbone as of April 15, 2007

Clev.

Boston

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Future ESnet Hub

ESnet Hub

5 ESnet 4 Backbone as of May 15, 2007

SNV

Clev.

Boston

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Clev.

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Future ESnet Hub

ESnet Hub

6 ESnet 4 Backbone as of June 20, 2007

Clev.

Boston

Houston

Kansas City

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Future ESnet Hub

ESnet Hub

7 ESnet 4 Backbone Target August 1, 2007

Clev.

Boston

Houston

Los Angeles

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Future ESnet Hub

ESnet Hub

8 ESnet 4 Backbone Target August 30, 2007

Clev.

Boston

Houston

Boise

Los Angeles

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Future ESnet Hub

ESnet Hub

9 ESnet 4 Backbone Target September 30, 2007

Clev.

Boston

Houston

Boise

Los Angeles

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Nashville

Future ESnet Hub

ESnet Hub

10

DC

ESnet 4 Backbone Target September 15, 2008

Houston 10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Los Angeles Nashville

Future ESnet Hub

ESnet Hub

11 ESnet4 Core networks 50-60 Gbps by 2009-2010 (10Gb/s circuits),

500-600 Gbps by 2011-2012 (100 Gb/s circuits)

Cle

vela

nd

Europe (GEANT)

Asia-Pacific

New York

Washington DC

CERN (30+ Gbps)

Albuquerque

Aus

tral

ia

San Diego

LA

Denver

South America (AMPATH)

South America (AMPATH)

Canada (CANARIE)

CERN (30+ Gbps) Canada (CANARIE) Asia Pacific

GLORIAD (Russia and

China)

Boise

Jacksonville

Tulsa

Boston

Science Data Network Core

IP Core

Aus

tral

ia

Core network fiber path is ~ 14,000 miles / 24,000 km

1625

mile

s / 2

545

km

2700 miles / 4300 km

Production IP core (10Gbps) SDN core (20-30-40-50 Gbps) MANs (20-60 Gbps) or backbone loops for site access International connections

IP core hubs

Primary DOE Labs SDN hubs

High speed cross-connects with Ineternet2/Abilene Possible hubs

12 Typical ESnet 4 Hub

M320 Router 7609 Switch

10G Performance Tester

Peering Router

Secure Term Server Power Controllers

OWAMP Measurement Device

13 ESnet 4 Factiods as of July 16, 2007

•  Installation to date: o  10 new 10Gb/s circuits o  ~10,000 Route Miles o  6 new hubs o  5 new routers 4 new switches

-  Total of 70 individual pieces of equipment shipped –  Over two and a half tons of electronics

o  15 round trip airline tickets for our install team -  About 60,000 miles traveled so far…. -  6 cities

–  5 Brazilian Bar-B-Qs/Grills sampled

14 ESnet Traffic Now Exceeding 2 PetaBytes/Month

2.4PBytes in May 2007

Tera

Byt

es

ESnet traffic historically has increased 10x every 47 months

1 PBytes in April 2006

15 ESnet Availability

“5 nines” (>99.995%) “3 nines” (>99.5%) “4 nines” (>99.95%)

Dually connected sites Note: These availability measures are only for ESnet infrastructure, they do not include site-related problems. Some sites, e.g. PNNL and LANL, provide circuits from the site to an ESnet hub, and therefore the ESnet-site demarc is at the ESnet hub (there is no ESnet equipment at the site. In this case, circuit outages between the ESnet equipment and the site are considered site issues and are not included in the ESnet availability metric.

16 OSCARS Overview

Path Computation •  Topology

•  Reachability •  Contraints

Scheduling •  AAA

•  Availability

Provisioning •  Signalling •  Security

•  Resiliency/Redundancy

OSCARS Guaranteed Bandwidth

Virtual Circuit Services

On-demand Secure Circuits and Advance Reservation System

17 OSCARS Status Update •  ESnet Centric Deployment

o  Prototype layer 3 (IP) guaranteed bandwidth virtual circuit service deployed in ESnet (1Q05)

o  Layer 2 (Ethernet VLAN) virtual circuit service under development

•  Inter-Domain Collaborative Efforts o  Terapaths

-  Inter-domain interoperability for layer 3 virtual circuits demonstrated (3Q06) -  Inter-domain interoperability for layer 2 virtual circuits under development

o  HOPI/DRAGON -  Inter-domain exchange of control messages demonstrated (1Q07) -  Initial integration of OSCARS and DRAGON has been successful (1Q07)

o  DICE -  First draft of topology exchange schema has been formalized (in collaboration with NMWG)

(2Q07), interoperability test scheduled for 3Q07 -  Drafts on reservation and signaling messages under discussion

o  UVA -  Integration of Token based authorization in OSCARS under discussion

18 ESnet Measurement Update

•  perfSONAR Services Deployed o  Measurement Archives serving Utilization information (Demo

later today) o  E2EMP serving end-to-end circuit status information o  Visualization tools

•  Bandwidth & Latency Measurement Points o  Deployed at ESnet4 hubs as they are configured

•  Working with the LHC community o  To define a set of perfSONAR services that can meet the

network measurement & monitoring needs of large distributed science application communities.