Slide 1

100 Gbps Services, Technologies, and Facilities: Basic Issues, Early Implementations, Demonstrations

Wendy Huntoon, Director Advanced Networking, (huntoon@psc.edu)Pittsburgh Supercomputing Center/3Rivers Optical Exchange

Abdella Battou, Interim Executive Director, (abattou@maxgigapop.net)

Joe Mambretti, Director, (j-mambretti@northwestern.edu)International Center for Advanced Internet Research (www.icair.org)Northwestern UniversityDirector, Metropolitan Research and Education Network (www.mren.org)Co-Director, StarLight (www.startap.net/starlight)

Quilt Annual MeetingPortland OregonSeptember 20111Need for Massive Additional Capacity Support for Capacity Much Beyond Aggregation of Millions of Small FlowsSupport for Extremely Large Individual Stream (Including End-To-End)Communications for Data Intensive (e.g., Petascale Science) Communications for Specialized Highly Distributed EnvironmentsEnvironments Directly Controlled By Edge Processes (Application Specific Network Services)Highly Controllable Science WorkflowsBegin Migration From Centralized NOCsScience Clouds (vs Consumer and Enterprise)Many New Applications and Services That Cannot Be Supported Today

Drivers for 100 Gbps Services2National Science Foundation Academic Research Infrastructure Program (ARI)Traditionally, This Program Has Funded Structures - e.g., Creating and Rehabbing Specialized Buildings and LaboratoriesCurrently, For the First Time, The Program Allows Communications Infrastructure To Be Included As StructureARIs Are Limited To One Per Campus, Requiring Local Organization CompetitionThe Funding Is Directed At Supporting Science Research Not General R&E Services

NSF Advanced Research Infrastructure Program3Four Worlds of 100 Gbps

100 Gbps Routing100 Gbps Ethernet100 Gbps OpticalOthere.g., Fiber Bundles, Interconnections, Control Planes, etc.4100 Gbps RoutingAvailable Today Based on Proprietary Technology Optimal Network Designs Place Such Devices At the Network Edge vs Network Core

100 Gbps Services: Routing5100 GigE L2 Switching

Standard: IEEE 802.3baTechnical changes Finalized In July 2009Formal Final Approval Took Place In July 2010Beta Products Available Q4 20101st Commercial Products End of Q4 2011Provides for a Rate of 103.125 Gbps

100 Gbps Services6100 Gbps Optical Switching

Standard: ITU G.709 v3 (ODU4 100G)ODU4/OTU4 Format -- Designed to Transport 100GbE (OTU4 = the ODU4 With FEC IncludedFormal Final Approval Took Place In Dec 2009Beta Products Available Today Ref: Demonstrations at SC101st Commercial Products Available End of Q3 2011

100 Gbps Services: WAN Side/Line Side 7100 Gbps Switch Design IssuesOptical/Electrical InterfacesData Plane ProcessingControl Plane Processing Switch FabricsModulation Techniques: e.g., Proprietary Dual Polarization Quadrature Phase Shift Keying (QPSK), a Format Used With 50GHz Channel Spacing (4 Signals In the Same Channel Spacing)PowerEtc.

100 Gbps Services: WAN Side/Line Side8100 GigE Physical Layer Standard (PHY) Objectives

Preserve 802.3 / Ethernet Frame Format Based On 802.3 MACPreserve Min/Max Frame Size of 802.3 StandardProvide PHY Specifications For Single-Mode Optical Fiber, Multi-Mode Optical Fiber (MMF), Copper Cables, Backplanes. Support Bit Error Ratio (BER) Better Than or Equal to 10 12 at the MAC/PLS Service InterfaceProvide Appropriate Support for Optical Transport Network (OTN) Standard

100 Gbps Services: Client Side 9Physical and Link Layer100G Long-Distance DWDM Transmission Framework The Framework Project Documents High Level System Objectives for Initial Implementations of 100G Long-Haul DWDM Transmission. It Identifies Transceiver Module Functional Architecture, and Decomposes That Architecture Into a Number of Technology Building Blocks. This Project Is Directed At Developing a Consensus Among a Critical Mass of Module and System Vendors On Requirements For Specific 100G technology Elements To Create Largest Possible Market for These Components. This Project Complements and Builds on Work Defining 100G Ethernet in the IEEE, and the Optical Transport Hierarchy (OTH) in the ITU-T.

Optical Internetworking Forum (OIF)100 Gbps Standards10100G Long Distance DWDM Integrated Photonics Receiver This Project Specifies Key Aspects of Integrated Receivers for Coherent DWDM Applications. Initially Targeting 100G PM-QPSK Applications, Project Intends to Remain Modulation Format and Data Rate Agnostic Whenever Practical to Maximize Applicability to Future Market Requirements. Key Aspects of Project Include Definition of: (1) Required Functionality. (2) High Speed Electrical Interfaces. (3) Low Speed Electrical Interfaces. (4) Optical Interfaces. (5) Mechanical Requirements. (6) Environmental Requirements. One Example: Optical Internetworking Forum100 Gbps Standards11100 Gbps Initiative PartnersPittsburgh Supercomputing Center3Rivers Optical ExchangeMid-Atlantic Cross RoadsMetropolitan Research and Education Network (MREN)StarLight International/National Communications ExchangeInternational Center for Advanced Internet Research (iCAIR), Northwestern UniversityUniversity of Illinois at ChicagoArgonne National LaboratoryFermi National Accelerator LaboratoryNational Center for Supercomputing ApplicationsCHI MAN Chicago Metro Area Optical Transport Facility for national Research Labs and StarLight (Multiple 100 Gbps)Illinois Wired/Wireless Infrastructure for Research and Education (I-WIRE)University of ChicagoEnergy Science Network 100 G Advanced Network InitiativeEnergy Science NetworkNASA Goddard Space Flight CenterCANARIE, SURFnetNational Oceanographic and Atmospheric Administration etc123ROX ARI ProjectRegional Network Renovation: Upgrading the 3ROX Virtual Research Environment Start date - 9/2010Duration - 36 monthsObjectivesUpgrade 3ROX DWDM infrastructure Upgrade Layer2/Layer2 infrastructure to be 100 GE capableProvide resource pool to researchers133ROX ARI StatusDWDM UpgradeCompleted in March 2011Features included:Increased lambda capacity - 4 lambdas per segment to up to 40 lambdas per segmentAdded ACM as a PoP on the ringReduced the number of transponders required to traverse the metro area Support 40 GE and 100GE lambdas

3ROX ARI Status - DWDM Ring

3ROX ARI Status100 GERFP out for 100 GE optical equipmentResponses due September 23 - significant interestLayer2/Layer3Minimum capability is to support 100 GE Internet2 connectionIdeally support switched 100 GESupport for OpenFlow optionalRFP to be released by the end of SeptemberResponses due 3 weeks after release

3ROX ARI StatusCreate Research Resource PoolEstablishes a pool of resources available to 3ROX associated researchersCurrent Resources10 GE transponders10 GE switch port and optics3ROX PoP co-location space and power - as availablePending100 GE transponders100 GE switch port and optic 3ROX ARI StatusProcessSubmit a brief proposal requesting resourcesReviewed by the 3ROX RRP Review committeeIf approved, resources allocated for a 6 to 12 month periodResearchers can request a time extension for longer projectsProgram GoalReduce the barrier for researchers to utilize network infrastructure, including 100 GEProvide mechanism for researchers to experiment with network resources before committing funds MAX ARI ProjectNSF 09-562 submitted 08/24/2009OIA Academic Research InfrastructureStarting Date : 05/03/10Duration 24 monthsObjectivesEnhance existing Research and Education metro network with 100G equipment to support data intensive science exploration, modeling, and discoveryUpgrade the 10G connection to Internet2 with 100GUpgrade the NGIX-East to 100G

19MAX 100G StatusPacket Optical Switching RFI issued on 4/6/2010Selected Fujitsu on 5/27/2010Acquired 6 Fujitsu 9500 Packet Optical on 9/20/2010Deployed them on 2/28/2011100G transponders expected end of September, 2011100G L2 Ethernet Switching RFI issued on 6/6/2011Internally Selected Brocade MLX-E on 08/22/2011Order for 4 100G ports waiting approval from NSF20MAX 100G Research Network

21E-science ApplicationsIt is the nature of science problems that they deal with a continuum of time and space parameters, which need to be quantized to adapt to available network throughput. For e-science experiments, it is never the question of how much throughput is needed to support an experiment, but what the accuracy of an experiment is given the network throughput. For example, a climate modeling experiment will be able to work with temperature and pressure gradients over a 3-D mesh which is ten times finer in each spatial dimension if the network throughput is improved by three orders of magnitude.

Thus, credible science experiments tend to, and in fact are designed to consume as much network throughput as possible

We better start working on the next higher rate : 1 Tbps interface !22IP Router Port Cost Improvement

Router ARouter BRouter CRouter DRouter ARouter BRouter CRouter DBefore: After:

Expensive 100Gbs router ports are reduced 50% Traffic is efficiently routed to appropriate destination (ex. A-C, vs. A-B-C)2310G consolidations

Any 10G waves along the same path could be aggregate into a 100G using OTN 24MREN Submitted ARI Proposal Early in 2010Award Announced In October 2010Award Granted To Metropolitan Research and Education Network/StarLight International/National Communications Exchange FacilityStarWave, A New Advanced Multi-100 Gbps Facility and Services Is Currently Being Designed It Will Be Implemented Within the StarLight International/National Communications Exchange FacilityThe Optical Components Will Be Implemented September 22-24, 2011Demonstration Showcases Planned for Oct, 2011, Nov 2011 (SC11, Seattle)

Metropolitan Research and Education Network (MREN) Multi-100 Gbps Facility25StarWave: A Multi-100 Gbps FacilityStarWave Is Being Funded To Provide Services To Support Large Scale Data Intensive Science Research InitiativesFacilities Components Will Include: An ITU G. 709 v3 Standards Based Optical Switch for WAN Services, Supporting Multiple 100 G ConnectionsAn IEEE 802.3ba Standards Based Client Side Switch, Supporting Multiple 100 G Connections, Multiple 10 G ConnectionsMultiple Other Components (e.g., Optical Fiber Interfaces, Measurement Servers, Test Servers26CA*net/Ciena/StarLight/iCAIR 100 Gbps Testbed Sept-Oct 2010

Source: CANARIE100 Gbps~850 Miles, 1368 Km27CA*net/Ciena/StarLight/iCAIR 100 Gbps Testbed Sept-Oct 2011

Source: CANARIE100 Gbps~850 Miles, 1368 Km28

29

30

DOE ESnet Advanced Networking Initiative: 100 GbpsSource : ESnet31Future International 100 Gbps Services Via Global Lambda Integrated FacilityAvailable Advanced Network ResourcesVisualization courtesy of Bob Patterson, NCSA; data compilation by Maxine Brown, UIC.www.glif.is

iCAIR is a co-founder of the GLIF, a consortium of institutions, organizations, consortia and country National Research & Education Networks who voluntarily share optical networking resources and expertise to develop the Global LambdaGrid for the advancement of scientific collaboration and discovery. 32t

Component of Next Gen Control Framework: Fenius GLIFDemonstrationsGlobal Lambda Grid WorkshopSC10HK GLIF Technical Workshop33SC11 Demonstration PlansConnect MAX and MREN to SCinet to support 100 GE demonstrationsBackbone connectivityUtilize existing backbone (ESnet, Internet2) to connect MAX and MREN to ScinetInvestigating dark fiber path between MAX and MREN for additional pathWould light with Fujitsu optical equipmentSCInet SCInet NOC, NASA, Ciena and LAC/iCAIR

100 GE WAN ConnectivitySeattleChicagoMcLeanWANWANNOCLA/iCAIR

CienaNASAS/ROpticalOpticalOpticalARI AwardsOrganizationTitle Awarded Amount University Corporation For Atmospheric ResearchThe Bi-State Optical Network (BiSON) Upgrade $ 430,000 University of HawaiiEnabling Hawaii to COMPETE: Seizing the Opportunity for Equitable Connectivity $ 9,838,151 Northwestern UniversityARI: Enhancing MREN Optical Communications Infrastructure for e-Science $ 1,374,132 University of Maryland College ParkRenovating Mid-Atlantic Crossroads Network Facilities $ 1,442,919 North Dakota State University FargoNorthern Tier Network--North Dakota and South Dakota Access Improvement $ 751,244 North Dakota State University FargoEnabling Inclusive Western States NSF Research via Next Generation Networking - the Northern Tier $ 1,157,498 University of New MexicoA Dedicated Advanced Science and Engineering Enterprise Network: The National Backbone Component $ 2,992,000 Rensselaer Polytechnic InstituteNYSERNet Research and Education network renovations $ 390,050 Carnegie-Mellon UniversityRegional Network Renovation: Upgrading the 3ROX Virtual Research Environment $ 1,535,012 Clemson University Research FoundationExpanding and Improving the C-Light Regional Optical Network $ 1,431,340 $ 21,342,346