Translight/PacificWaveTranslight/PacificWave

John SilvesterJohn SilvesterExecutive Director of Center for Scholarly TechnologyExecutive Director of Center for Scholarly TechnologyProfessor or Computer Engineering, USCProfessor or Computer Engineering, USCChair, CENICChair, CENIC

Briefing to NSF-OISE, April 28, 2006

NSF OCI-IRNC Cooperative Agreement [#0441119]NSF OCI-IRNC Cooperative Agreement [#0441119]

Translight/PacificWave (TL/PW)Translight/PacificWave (TL/PW) Cooperative Agreement [#0441119] under NSF Cooperative Agreement [#0441119] under NSF

IRNC Program IRNC Program PI John Silvester (USC); Co-PI Ron Johnson PI John Silvester (USC); Co-PI Ron Johnson

(UW)(UW) Other Key PartnersOther Key Partners

CENIC, Jim DolgonasCENIC, Jim Dolgonas PNWG, Jacqueline BrownPNWG, Jacqueline Brown University of Hawaii, David LassnerUniversity of Hawaii, David Lassner AARnet, George McLaughlinAARnet, George McLaughlin

Approx $1M per year for 5 yearsApprox $1M per year for 5 years Awarded March 1, 2005Awarded March 1, 2005

TL/PW Primary ObjectivesTL/PW Primary Objectives Facilitate international R&E connections on US Pacific Facilitate international R&E connections on US Pacific

CoastCoast Assist in the termination (equipment, backhaul) of Assist in the termination (equipment, backhaul) of

AARnet SXTransport links to Hawaii, Seattle and Los AARnet SXTransport links to Hawaii, Seattle and Los AngelesAngeles

Assist in build-out of Hawaii connectivity to TelescopesAssist in build-out of Hawaii connectivity to Telescopes Further develop PacificWave exchange capabilitiesFurther develop PacificWave exchange capabilities Assist in operation of IEEAF link Tokyo-SeattleAssist in operation of IEEAF link Tokyo-Seattle Assist in operation and support of other International Assist in operation and support of other International

R&E networks participating in PWR&E networks participating in PW Develop and operate advanced capabilities to support Develop and operate advanced capabilities to support

optical interconnect and exchange needs of R&E optical interconnect and exchange needs of R&E networksnetworks

What is Pacific Wave?What is Pacific Wave?

Pacific Wave is a state-of-the-art international Pacific Wave is a state-of-the-art international peering facility designed to serve advanced peering facility designed to serve advanced research and education.research and education.

Primary focus is interconnectivity among US Primary focus is interconnectivity among US R&E networks and International R&E networks R&E networks and International R&E networks from the Pacific Rimfrom the Pacific Rim

One of the objectives was the desire for all One of the objectives was the desire for all participants to be able to peer with each other participants to be able to peer with each other which led to the “distributed exchange” designwhich led to the “distributed exchange” design

Context of Pacific WaveContext of Pacific Wave

The US States on the Pacific Coast [Alaska, The US States on the Pacific Coast [Alaska, Washington, Oregon, California, Hawaii] have Washington, Oregon, California, Hawaii] have collaborated in networking for several years. (e.g. US collaborated in networking for several years. (e.g. US Pacific Consortium is an affiliate of APAN)Pacific Consortium is an affiliate of APAN)

Seattle, Los Angeles, and Sunnyvale are “natural” places Seattle, Los Angeles, and Sunnyvale are “natural” places for exchange pointsfor exchange points

Many international links enter the US through Many international links enter the US through Washington and California (and to a certain extent Washington and California (and to a certain extent Hawaii although they tend to remain local there)Hawaii although they tend to remain local there)

Led to the development of Pacific Wave – a collaboration Led to the development of Pacific Wave – a collaboration between CENIC and PNWG to build out exchange between CENIC and PNWG to build out exchange capabilities on West Coast.capabilities on West Coast.

Who Operates Pacific Wave?Who Operates Pacific Wave?A joint project of CENIC and Pacific Northwest Gigapop

In collaboration with University of Southern California and University of Washington

Pacific Wave Layer 2 ExchangePacific Wave Layer 2 Exchange Extensible peering exchange to allow layer 3 networks to peer easily Extensible peering exchange to allow layer 3 networks to peer easily

over a layer 2 fabricover a layer 2 fabric Nodes (currently) in Seattle, Sunnyvale, and Los Angeles. Connected Nodes (currently) in Seattle, Sunnyvale, and Los Angeles. Connected

by a 10GbE wave provisioned over National LambdaRail (2,241 by a 10GbE wave provisioned over National LambdaRail (2,241 kilometers)kilometers)

AUP freeAUP free Supports IPv4 and IPv6; multicast enabledSupports IPv4 and IPv6; multicast enabled Provides 24x7 NOC support.Provides 24x7 NOC support. Priced consistently from node to node Priced consistently from node to node Allows participants to self-select their peeringAllows participants to self-select their peering Allows participants to connect to one-location and access participants Allows participants to connect to one-location and access participants

at all Pacific Wave nodesat all Pacific Wave nodes Supports advance applicationsSupports advance applications Welcomes any research or development network that can meet the Welcomes any research or development network that can meet the

minimum network configuration requirements (connect with a router; minimum network configuration requirements (connect with a router; support BGP; support jumbo-frames)support BGP; support jumbo-frames)

International ParticipationInternational Participation

““Layer 3” R&E Networks and Pacific WaveLayer 3” R&E Networks and Pacific Wave

AARNET

GEMNET

GLORIAD

IEEAF

KREONET

MIMOS (Malaysia)

QATAR

SINET

SINGAREN

TANET/TWAREN

TRANSPAC2

ABILENE (Internet2)

CALREN (HPR)

DREN ESNET

NASA

NLR-Packetnet

PNWG

Pacific Wave

CA*NET4

CLARACUDI

PacificWave SeattlePacificWave Seattle

Seattle WestinCISCO 6509

NLR Lambda to LAX (via Sunnyvale)

Abilene

AARNet

CA*NET4

GEMnet

SingAREN

TANET2/TWARENATTBI/Comcast

DREN

KREONet2

ESnet Microsoft

PNWGP

NLR Lambda to STARLIGHT

WIDE/TLEX(IEEAF)

NLR-Packetnet

Does not show “lightpath” connections

1 Gigabit

10 Gigabit

Pacific Wave - SunnyvalePacific Wave - Sunnyvale

NLR TO SEATTLE PWAVE

L3 SVL6509

CENIC SVL HPR

NLR TO LA PWAVE

PAIX

ESNET

Does not show “lightpath” connections

1 Gigabit

10 Gigabit

Pacific Wave - Los AngelesPacific Wave - Los Angeles

1 Wilshire6509

600 W 7th6509

818 W 7th6509

CalREN-HPR

10 GE NLR LambdaTo Seattle PWave(via Sunnyvale)

Abilene

TWAREN

TRANSPAC2

Los Nettos

Qatar Foundation

NII/SINET

AARNet

(3)Singaren

Mimos Berhad(Malaysia)

Cinegrid NLRPacketnet

Does not show “lightpath” connections

1 Gigabit

10 Gigabit

Network TopologyNetwork Topology

The VLAN ArchitectureThe VLAN Architecture

IRNC TL/PW Pacific ConnectionsIRNC TL/PW Pacific Connections

CA*Net4 POP

TλEX Tokyo PW-Seattle

AARnet POP Sydney

Hawaii

OahuPW-LA

CLARA, CUDI POPs (Tijuana)

AARnet-SX Transport

IEEAF Link

TL-PW – West Coast DetailTL-PW – West Coast Detail

AARNet-SX(N)To Kahe Point (HI)

Los Osos(CA)

SLO(CA)

Hillsboro(OR)

Portland (OR)

PW-SeattleNLR link

PW-Los Angeles

Pacific WaveN-S link

AARNet-SX(S)To Spencer Beach (HI)

SEA

LAX

Santa Barbara(CA)

SNV

(NLR)

(CalREN-XD)

Optics to carrynew lambda

over CalREN-XD from SLO to LA

TL/PW – Hawaii DetailTL/PW – Hawaii Detail

SCCNOC-192

SCCNOC-192

AARNet-SX(S)To Sydney

Spencer Beach(Big Island)

Mauna KeaMKOCN (Summit)

Hilo

AARNet-SX(S)To Los Osos

Waimea

Kahe Point(Oahu)

Hawaii Gigapop(Oahu)AARNet-

SX(S)To Sydney

AARNet-SX(N)To Portland

Mauna Kea Observatories Communication Network

CSO Gemini IRTFJCMT SMA UKIRT UH-2.2 UH-0.6 VLBACFHT Keck

CSO

Gemini

JAC

Subaru

SMA

CFHT

Keck

VLBA

UH IfA Hilo

UH-Managed Shared Network for All Observatories

Verizon Summit

Verizon Hilo

VerizonHumuula

VerizonWaimea

U N I V E R S I T YU N I V E R S I T Y

UH Hilo Campus

Subaru

DS3

Leased Private Links

Carrier Infrastructurefor leased shared & private links

Leased DS3 (shared)

UH managed fiber links

56kb

Hilo Base Facilities

Waimea Base Facilities

Proposed New GbE

Status of TL/PW as of April 2006Status of TL/PW as of April 2006

Engineering and technical support through PNWG Engineering and technical support through PNWG and CENIC for all International connections into and CENIC for all International connections into Pacific WavePacific Wave

Northern AARnet LinkNorthern AARnet Link Operational and in useOperational and in use

Southern AARnet LinkSouthern AARnet Link Backhaul from Los Osos through San Luis Obispo to PW-Backhaul from Los Osos through San Luis Obispo to PW-

LA over CALREN as OC-192 in placeLA over CALREN as OC-192 in place ETE equipment for HI and LA on order (arriving as we ETE equipment for HI and LA on order (arriving as we

speak).speak). Local connectivity in Hawaii – in processLocal connectivity in Hawaii – in process Expected operation (test mode) end of MayExpected operation (test mode) end of May

Changing Requirements for Changing Requirements for ExchangesExchanges

The Exchange points need to provide a wider The Exchange points need to provide a wider array of services:array of services: Layer 3 Interconnect (routed)Layer 3 Interconnect (routed) Layer 3 Interconnect (over layer 2 switch)Layer 3 Interconnect (over layer 2 switch) Layer 2 Interconnect (Ethernet switching)Layer 2 Interconnect (Ethernet switching) Layer 1 Interconnect (wave switching)Layer 1 Interconnect (wave switching) Layer 0 Interconnect (fiber interconnect)Layer 0 Interconnect (fiber interconnect)

PacificWave is moving in this direction to allow PacificWave is moving in this direction to allow us to continue to participate in the world of us to continue to participate in the world of Global R&E networkingGlobal R&E networking

Pacific Wave Lightpath InterconnectPacific Wave Lightpath Interconnect

Primary focus has been on Layer 3 interconnectPrimary focus has been on Layer 3 interconnect There is a lot of interest in Layer 2 (“lightpaths”) There is a lot of interest in Layer 2 (“lightpaths”)

and Layer 1 (“waves” and “lambdas”) and Layer 1 (“waves” and “lambdas”) interconnect. [GLIF, etc]interconnect. [GLIF, etc]

SXTransport(S) will be configured as 8x1GESXTransport(S) will be configured as 8x1GE PW is deploying additional OC192 from LA to PW is deploying additional OC192 from LA to

SEA configured as 8x1GE to allow interconnect SEA configured as 8x1GE to allow interconnect for AARnet and other nets as demand grows.for AARnet and other nets as demand grows.

Also plans in place to extend PW connectivity to Also plans in place to extend PW connectivity to Starlight (Chicago) to further enhance layer 3 Starlight (Chicago) to further enhance layer 3 and lightpath interconnect capabilityand lightpath interconnect capability

Pacific Wave Lightpath ExchangePacific Wave Lightpath ExchangeTo Canarie

NLR To Chicago and Starlight

San Diego

NLR To Phoenix

IEEAF to TλEX

AARnet to AU, via HI

HOPI

HOPI

SEA

SNY

LA

Lightpath Networks and PWLightpath Networks and PW

Others in the planning stages

TRANSPAC (?)

AARNET (S)

(to TλEX) IEEAF

CA*NET4

NLR Waves

NLR (Layer 2)

CALREN-XD

HOPI (Internet2)

PNWG

Pacific Wave

AARNet S PathPW Lightpath Interconnect

Abilene

TwarenSingaren

PW-L3X:10GLAX-SNY

PW-L2X (8x1G)LAX-SNY-SEA

AARnetRouter

PW 15454

CalRENRouter

6509818

6509

6509

PW-L3X-LA

to SNY, SEA

CENIC 15808

to SLO

to SD

CENIC 15808

CENIC 15808

NLR Packetnet

Some Current ProjectsSome Current Projects

ActivitiesActivities

Basically all International R&E traffic to Basically all International R&E traffic to Pacific Rim flows through PWPacific Rim flows through PW

Recent HighlightsRecent Highlights iGRIDiGRID SC’05SC’05 PRAGMAPRAGMA APANAPAN INTERNET2 meetingsINTERNET2 meetings

ExamplesExamples January 2005: Huygens Titan probe data transfer: January 2005: Huygens Titan probe data transfer:

3Tbytes of satellite data distributed and collated for 3Tbytes of satellite data distributed and collated for analysis to locations in Amsterdam and Australiaanalysis to locations in Amsterdam and Australia

January 2005: Tele-microscopy demo between January 2005: Tele-microscopy demo between University of Hawaii and UCSD (Lariat network)University of Hawaii and UCSD (Lariat network)

January 2005: PTC Honolulu 270Mbps interactive HD January 2005: PTC Honolulu 270Mbps interactive HD video conferencing with Australia and Seattlevideo conferencing with Australia and Seattle

January 2005: Seattle—Tokyo, uncompressed HDTV 1.5 January 2005: Seattle—Tokyo, uncompressed HDTV 1.5 Gbps Smarr HD lecture between Seattle and JGN Gbps Smarr HD lecture between Seattle and JGN meeting meeting

June 2005: Chicago-Seattle-UCSD OptiPuter HD June 2005: Chicago-Seattle-UCSD OptiPuter HD permanent interactive uncompressed HDTV video-permanent interactive uncompressed HDTV video-conference linkconference link

Examples – iGRID05Examples – iGRID05 4k interactive HD between Tokyo & SD4k interactive HD between Tokyo & SD Concurrent n-way real-time low-latency uncompressed Concurrent n-way real-time low-latency uncompressed

1080 line (ie >1gb/s per stream) HDTV teleconferencing 1080 line (ie >1gb/s per stream) HDTV teleconferencing between Australia, Japan, Korea, Ann Arbor, Seattle between Australia, Japan, Korea, Ann Arbor, Seattle and San Diego and San Diego

100 megapixel remote viz. Illinois-SD100 megapixel remote viz. Illinois-SD Real-time flat-panel stereo, remotely rendered Real-time flat-panel stereo, remotely rendered Clusters in Chicago & Amsterdam loosely coupled Clusters in Chicago & Amsterdam loosely coupled

compute demo: performance = to single machine room compute demo: performance = to single machine room alternative!!alternative!!

Neptune/‘Looking’ real-time HDTV from 2km below the Neptune/‘Looking’ real-time HDTV from 2km below the surface showing volcanic vents in the northeast pacificsurface showing volcanic vents in the northeast pacific

ExamplesExamples

November 14-17, 2005: SC05 Seattle. About 50 10gb/s November 14-17, 2005: SC05 Seattle. About 50 10gb/s waves, petabytes of storage, leading edge visualization waves, petabytes of storage, leading edge visualization systems and teraflops of computing assembled for a systems and teraflops of computing assembled for a week computational sciences etc conference for this. week computational sciences etc conference for this.

‘‘Persistent’ OptIPuterPersistent’ OptIPuter Wide Area Storage GRIDsWide Area Storage GRIDs The Next Disaster Recovery Architectures that really The Next Disaster Recovery Architectures that really

work!? work!? Neptune Canada ‘in the water’ Neptune Canada ‘in the water’

‘Dry’ Headend – Nodes & Access points

“Neptune” Ocean Observatories

Recent Stereo VideoRecent Stereo Video

I2 Spring MM from Stanford Medical School (Margaret I2 Spring MM from Stanford Medical School (Margaret Krebs, Testbed Mgr, Stanford University Medical Media Krebs, Testbed Mgr, Stanford University Medical Media and Information Technologies) ran a stereo video and Information Technologies) ran a stereo video program test with CSIRO in Sydney on March 30th via  program test with CSIRO in Sydney on March 30th via  Abilene and SX.  Abilene and SX. 

They ran about a 90Mbps stream for the  test.  They ran about a 90Mbps stream for the  test.  No problems through AARnet N and across AARnet.No problems through AARnet N and across AARnet.

iGrid 2005, San Diego Sept. iGrid 2005, San Diego Sept. 20052005

iGrid 2005, Over 10 Gbps bidirectional traffic coexisted iGrid 2005, Over 10 Gbps bidirectional traffic coexisted with production exchange traffic without detrimentwith production exchange traffic without detriment

4k line interactive Super-HD between Keio University and UCSD4k line interactive Super-HD between Keio University and UCSD Live HD from the sea floor 100 miles off the Pacific CoastLive HD from the sea floor 100 miles off the Pacific Coast N-way uncompressed multicast HD video conferencingN-way uncompressed multicast HD video conferencing

ResearchChannel N-Way HD ResearchChannel N-Way HD Multicast Video ConferencingMulticast Video Conferencing

Does it Really Perform in Does it Really Perform in Practice? …Yes!Practice? …Yes!

7.5 Gbps of unicast 7.5 Gbps of unicast HDTV over IP traffic HDTV over IP traffic (10 750 Mbps flows)(10 750 Mbps flows)

3 Gbps of multicast 3 Gbps of multicast replicated in Seattle replicated in Seattle switch node using switch node using PIM snoopingPIM snooping

Traffic test performed Traffic test performed in Cisco POC lab in Cisco POC lab before the eventbefore the event

An Amazing Feat at SC05An Amazing Feat at SC05

47 10 Gbps Lambdas to be 47 10 Gbps Lambdas to be ExactExact

HDTV Conferencing Spanning Two HDTV Conferencing Spanning Two Oceans Enabled by Pacific Wave Oceans Enabled by Pacific Wave

Traffic Flows at SupercomputingTraffic Flows at Supercomputing

Among other things, 5-way multicast Among other things, 5-way multicast replication of 3 Gbps, over 12 Gbps total replication of 3 Gbps, over 12 Gbps total traffic across the switch backplanestraffic across the switch backplanes

Thank you

John Silvester, silvester@usc.edu