1

ESnet UpdateWinter 2008

Joint Techs Workshop

Joe BurresciaESnet General Manager

January 21, 2008

Energy Sciences NetworkLawrence Berkeley National Laboratory

Networking for the Future of Science

2

TWC

SNLL

YUCCA MT

BECHTEL-NV

PNNLLIGO

INEEL

LANL

SNLAAlliedSignal

PANTEX

ARM

KCP

NOAA

OSTI ORAU

SRS

JLAB

PPPL

Lab DCOffices

MIT

ANL

BNL

FNALAMES

NR

EL

LLNL

GA

DOE-ALB

OSC GTNNNSA

International (high speed)10 Gb/s SDN core10G/s IP core2.5 Gb/s IP coreMAN rings (≥ 10 G/s)Lab supplied linksOC12 ATM (622 Mb/s)OC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less

NNSA Sponsored (12)Joint Sponsored (3)

Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)

42 end user sites

SINet (Japan)Russia (BINP)CA*net4

FranceGLORIAD (Russia, China)Korea (Kreonet2

Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren

ESnet IP core: Packet over

SONET Optical Ring and Hubs

ELP

DC

commercial peering points

MAE-E

PAIX-PAEquinix, etc.

PN

WG

Po

P/

PA

cifi

cWav

e

ESnet 3 with Sites and Peers (Early 2007)

ESnet core hubs IP

Abilene high-speed peering points with Internet2/Abilene

Abilene

Ab

ilen

e

CERN(USLHCnet

DOE+CERN funded)

GÉANT - France, Germany, Italy, UK, etc

NYC

Starlight

SNV

Ab

ilene

JGI

LBNL

SLACNERSC

SNV SDN

SDSC

Equinix

SNV

ALB

ORNL

CHI

MRENNetherlandsStarTapTaiwan (TANet2, ASCC)

NA

SA

Am

es

AU

AU

SEA

CH

I-SL M

AN

LA

NA

bile

ne

Specific R&E network peers

Other R&E peering points

UNM

MAXGPoP

AMPATH(S. America)

AMPATH(S. America)

ES

net

Scie

nce D

ata

Netw

ork

(S

DN

) core

R&Enetworks

Office Of Science Sponsored (22)

ATL

NSF/IRNCfunded

Equinix

3

ESnet 3 Backbone as of January 1, 2007

Sunnyvale

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)MAN rings (≥ 10 G/s)Lab supplied links

Seattle

San Diego Albuquerque

El Paso

Chicago New York

City

Washington DC

Atlanta

Future ESnet Hub

ESnet Hub

4

ESnet 4 Backbone as of April 15, 2007

Clev.

Boston

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Future ESnet Hub

ESnet Hub

5

ESnet 4 Backbone as of May 15, 2007

SNV

Clev.

Boston

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Clev.

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Future ESnet Hub

ESnet Hub

6

ESnet 4 Backbone as of June 20, 2007

Clev.

Boston

Houston

Kansas City

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Future ESnet Hub

ESnet Hub

7

ESnet 4 Backbone August 1, 2007 (Last JT meeting at FNAL)

Clev.

Boston

Houston

Los Angeles

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Future ESnet Hub

ESnet Hub

8

ESnet 4 Backbone September 30, 2007

Clev.

Boston

Houston

Boise

Los Angeles

10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Future ESnet Hub

ESnet Hub

9

ESnet 4 Backbone December 2007

Clev.

Boston

Houston

Boise

Los Angeles

10 Gb/s SDN core (NLR)2.5 Gb/s IP Tail (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

New York City

Washington DC

Atlanta

Denver

Nashville

Future ESnet Hub

ESnet Hub

Chicago

10

DC

ESnet 4 Backbone December, 2008

Houston 10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Los Angeles

Nashville

Future ESnet Hub

ESnet Hub

X2

X2

X2

X2

X2

X2

X2

LVK

SNLL

YUCCA MT

BECHTEL-NV

PNNLLIGO

LANL

SNLAAlliedSignal

PANTEX

ARM

KCP

NOAA

OSTI

ORAU

SRS

JLAB

PPPL

Lab DCOffices

MIT/PSFC

BNL

AMES

NREL

LLNL

GA

DOE-ALB

DOE GTNNNSA

NNSA Sponsored (13+)Joint Sponsored (3)Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)

~45 end user sites

SINet (Japan)Russia (BINP)CA*net4

FranceGLORIAD (Russia, China)Korea (Kreonet2

Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren

ELPA

WA

SH

commercial peering points

PAIX-PAEquinix, etc.

ESnet Provides Global High-Speed Internet Connectivity for DOE Facilities and Collaborators (12/2007)

ESnet core hubs

CERN(USLHCnet:

DOE+CERN funded)

GÉANT - France, Germany, Italy, UK, etc

NEWY

SUNN

Ab

ilen

e

JGI

LBNL

SLACNERSC

SNV1

SDSC

Equinix

ALBU

OR

NL

CHIC

MRENStarTapTaiwan (TANet2, ASCC)

NA

SA

Am

es

AU

AU

SEA

CH

I-SL

Specific R&E network peers

UNM

MAXGPoPNLR

AMPATH(S. America)

AMPATH(S. America)

R&Enetworks

Office Of Science Sponsored (22)

ATLA

NSF/IRNCfunded

IARC

Pac

Wav

e

KAREN/REANNZODN Japan Telecom AmericaNLR-PacketnetAbilene/I2

KAREN / REANNZInternet2SINGARENODN Japan Telecom America

NETL

ANLFNAL

Starlight

USLH

CN

et

NLR

International (1-10 Gb/s)10 Gb/s SDN core (I2, NLR)10Gb/s IP coreMAN rings (≥ 10 Gb/s)Lab supplied linksOC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less

Salt Lake

PacWave

Ab

ilen

e

Equinix DENV

DOE

SUNN

NASH

Geography isonly representational

Inte

rnet

2N

YS

ER

Net

MA

N L

AN

Other R&E peering points

US

HL

CN

et

to G

ÉA

NT

INL

12ESnet4Core networks 50-60 Gbps by 2009-2010 (10Gb/s circuits),

500-600 Gbps by 2011-2012 (100 Gb/s circuits)

Cle

vela

nd

Europe(GEANT)

Asia-Pacific

New York

Chicago

Washington DC

Atl

anta

CERN (30+ Gbps)

Seattle

Albuquerque

Au

str

ali

a

San Diego

LA

Denver

South America(AMPATH)

South America(AMPATH)

Canada(CANARIE)

CERN (30+ Gbps)Canada(CANARIE)

Asi

a-Pac

ific

Asia Pacific

GLORIAD (Russia and

China)

Boise

HoustonJacksonville

Tulsa

Boston

Science Data Network Core

IP Core

Kansa

s

City

Au

str

ali

a

Core network fiber path is~ 14,000 miles / 24,000 km

162

5 m

iles

/ 2

545

km

2700 miles / 4300 km

Sunnyvale

Production IP core (10Gbps)

SDN core (20-30-40-50 Gbps)

MANs (20-60 Gbps) or backbone loops for site access

International connections

IP core hubs

Primary DOE LabsSDN hubs

High speed cross-connectswith Ineternet2/AbilenePossible hubs

USLHCNet

13A Tail of Two ESnet4 Hubs

MX960 Switch

T320 Router

6509 Switch

T320 Routers

Sunnyvale Ca Hub Chicago HubESnet’s SDN backbone is implemented with Layer2 switches; Cisco 6509s and Juniper MX960s each present their own unique challenges.

14ESnet 4 Factoids as of January 21, 2008

• ESnet4 installation to date:o 32 new 10Gb/s backbone circuits

- Over 3 times the number from last JT meeting

o 20,284 10Gb/s backbone Route Miles - More than doubled from last JT meeting

o 10 new hubs- Since last meeting

– Seattle– Sunnyvale– Nashville

o 7 new routers 4 new switcheso Chicago MAN now connected to Level3 POP

- 2 x 10GE to ANL

- 2 x 10GE to FNAL

- 3 x 10GE to Starlight

15ESnet Traffic Continues to Exceed 2 PetaBytes/Month

Bytes Accepted

0.00E+00

5.00E+14

1.00E+15

1.50E+15

2.00E+15

2.50E+15

3.00E+15

Jan,

00

Apr, 0

0

Jul, 0

0

Oct, 0

0

Jan,

01

Apr, 0

1

Jul,

01

Oct, 0

1

Jan,

02

Apr, 0

2

Jul, 0

2

Oct, 0

2

Jan,

03

Apr, 0

3

Jul, 0

3

Oct, 0

3

Jan,

04

Apr, 0

4

Jul, 0

4

Oct, 0

4

Jan,

05

Apr, 0

5

Jul, 0

5

Oct, 0

5

Jan,

06

Apr, 0

6

Jul, 0

6

Oct, 0

6

Jan,

07

Apr, 0

7

Jul, 0

7

Oct, 0

7

2.7 PBytes in July 2007

1 PBytes in April 2006

ESnet traffic historically has increased 10x every 47 months

Overall traffic tracks the very large

science use of the network

When A Few Large Data Sources/Sinks Dominate Trafficit is Not Surprising that Overall Network Usage Follows the

Patterns of the Very Large Users - This Trend Will Reverse in the Next Few Weeks as the Next Round of LHC Data Challenges Kicks Off

17ESnet Continues to be Highly Reliable; Even During the Transition

ESnet Availability 2/2007 through 1/2008

0

200

400

600

800

1000

1200

1400

1600

1800

AN

L 1

00

.00

0

OR

NL

10

0.0

00

SL

AC

10

0.0

00

FN

AL

99

.99

9

LIG

O 9

9.9

98

LL

NL

99

.99

8

NE

RS

C 9

9.9

98

PN

NL

99

.99

8

DO

E-G

TN

99

.99

7

SN

LL

99

.99

7

LB

L 9

9.9

96

MS

RI 9

9.9

94

LL

NL

-DC

99

.99

1

NS

TE

C 9

9.9

91

LA

NL

-DC

99

.99

0

JG

I 99

.98

8

IAR

C 9

9.9

85

PP

PL

99

.98

5

JL

ab

99

.98

4

DO

E-A

LB

99

.97

3

LA

NL

99

.97

2

SN

LA

99

.97

1

Pa

nte

x 9

9.9

67

BN

L 9

9.9

66

NR

EL

99

.96

5

MIT

99

.94

7

DO

E-N

NS

A 9

9.9

17

Yu

cc

a 9

9.9

17

GA

99

.91

6

INL

99

.90

9

Be

ch

tel 9

9.8

85

KC

P 9

9.8

71

Y1

2 9

9.8

63

BJ

C 9

9.8

62

OR

AU

99

.85

7

Am

es

-La

b 9

9.8

52

OS

TI 9

9.8

51

NO

AA

99

.75

6

La

mo

nt

99

.75

4

SR

S 9

9.7

04

Ou

tag

e M

inu

tes

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Nov

Dec

Jan

“5 nines” (>99.995%) “3 nines” (>99.5%)“4 nines” (>99.95%)

Dually connected sites Note: These availability measures are only for ESnet infrastructure, they do not include site-related problems. Some sites, e.g. PNNL and LANL, provide circuits from the site to an ESnet hub, and therefore the ESnet-site demarc is at the ESnet hub (there is no ESnet equipment at the site. In this case, circuit outages between the ESnet equipment and the site are considered site issues and are not included in the ESnet availability metric.

18OSCARS Overview

Path Computation• Topology

• Reachability• Contraints

Scheduling• AAA

• Availability

Provisioning• Signalling• Security

• Resiliency/Redundancy

OSCARSGuaranteedBandwidth

Virtual Circuit Services

On-demand Secure Circuits and Advance Reservation System

19OSCARS Status Update

• ESnet Centric Deploymento Prototype layer 3 (IP) guaranteed bandwidth virtual circuit service deployed in ESnet

(1Q05)o Prototype layer 2 (Ethernet VLAN) virtual circuit service deployed in ESnet (3Q07)

• Inter-Domain Collaborative Effortso Terapaths (BNL)

- Inter-domain interoperability for layer 3 virtual circuits demonstrated (3Q06)- Inter-domain interoperability for layer 2 virtual circuits demonstrated at SC07 (4Q07)

o LambdaStation (FNAL)- Inter-domain interoperability for layer 2 virtual circuits demonstrated at SC07 (4Q07)

o HOPI/DRAGON- Inter-domain exchange of control messages demonstrated (1Q07)- Integration of OSCARS and DRAGON has been successful (1Q07)

o DICE- First draft of topology exchange schema has been formalized (in collaboration with NMWG)

(2Q07), interoperability test demonstrated 3Q07- Initial implementation of reservation and signaling messages demonstrated at SC07 (4Q07)

o UVA- Integration of Token based authorization in OSCARS under testing

o Nortel- Topology exchange demonstrated successfully 3Q07 - Inter-domain interoperability for layer 2 virtual circuits demonstrated at SC07 (4Q07)

20

• ESneto About 1/3 of the 10GE bandwidth test platforms & 1/2 of the

latency test platforms for ESnet 4 have been deployed.- 10GE test systems are being used extensively for acceptance

testing and debugging- Structured & ad-hoc external testing capabilities have not been

enabled yet.- Clocking issues at a couple POPS are not resolved.

o Work is progressing on revamping the ESnet statistics collection, management & publication systems

- ESxSNMP & TSDB & PerfSONAR Measurement Archive (MA)- PerfSONAR TS & OSCARS Topology DB- NetInfo being restructured to be PerfSONAR based

• LHC and PerfSONARo PerfSONAR based network measurement solutions for the

Tier 1/Tier 2 community are nearing completion.o A proposal from DANTE to deploy a perfSONAR based

network measurement service across the LHCOPN at all Tier1 sites is being evaluated by the Tier 1 centers

Network Measurement Update

21

End