SC08 Routing Infrastructure

Thomas Hutton - San Diego Supercomputer Center

Brent Sweeny - Indiana University/Global NOC

Routing Team SC08Key Network Engineers from leading

R&E Networks and Sites

• Linda Winkler (lead, ANL)• Eli Dart (ESnet)• Pieter de Boer (SARA)• Tom Hutton (SDSC/UCSD)• Andrew Lee (IU/GRNOC)• Craig Leres (LBL) • Corby Schmitz (ANL) • Brent Sweeny (IU/GRNOC)• JP Velders (uVa) • Alan Verlo (UIC)

SC08 Switching/Routing Objectives

• IPV4 and IPV6 in hardware everywhere• IPV4 Multicast in hardware everywhere• Jumbo frame support on showfloor• L3 peering with Internet2, ESnet, NLR, LEARN• L2 services from NLR, Internet2, ESnet• Commodity Internet

– Diverse providers (Qwest, LEARN)– Deliver during WAN staging (Sep22-26)– In-service until 11-21-08 noon

Goals/Challenges

• L1/L2 service offerings

• Utilizing ACC Switches/Fiber for Conference Routers

• Staging Design needs to evolve into Show network without disruption

• Large number of external WAN connections.– 18 10Gig

• 10 NLR provided• 8 Internet 2 provided

– 1 40Gig– 2 1GE Commodity Internet

• 1 Level 3 Networks• 1 The University of Texas System

Network Architecture• SCinet designed like a large metropolitan

area• Nationwide WAN connections to two

interconnected Redundant CORE Centers• Redundantly connected to 5 DNOC’s

(distributed city POP’s)• Separate conference network to support

CORE and conference.

Network Architecture

WAN

Added Fibers for Staging

Redundant CORE routing and switching

Core-rtr-1Cisco CRS-1

Core-Rtr-2Juniper T1600

• T1600 supports 100G per slot capacity

• Carrier Class (NSR, ISSU)

– 1Gig, 10Gig, 100Gig Ethernet and Sonet Interfaces• Scalability

– Extensible, Network Virtualization– Over 2,000,000 network routes

• Security– Unified Access Control– Layer 2-4 Policy Enforcement– Control Plane Protection (rate-limiting and ACLs)

Core-Switch-1Force10 E1200

• Switch Fabric Capacity– 1.68 Tbps

• Full-Mesh Forwarding Capacity– 1 Bpps

• Backplane Capacity– 5 Tbps

• I/O Line Card Slots– 14

• Line-rate GbE (Line-rate TeraScale Line Cards)– 672

• Total GbE (High Density TeraScale Line Cards)– 1260

• can switch or route (SCinet is pure layer2)

Core-Switch-2Foundry BigIron RX-16

• HighSpeed Layer 2 Switching

• System redundancy (switch, management and power) across all BigIron RX chassis—BigIron RX-4, RX-8, RX-16 and RX-32

• Interchangeable half-height line modules reduce sparing costs

• Advanced virtual output queue (VoQ) design eliminates head of line blocking and provides scalable quality of service

• High-availability design features redundant and hot-pluggable hardware, hitless Layer 2 software upgrades

• Advanced non-blocking Clos fabric features adaptive self-routing with graceful system degradation in the event of two or more module failures

• Embedded sFlow per port supports scalable hardware-based traffic monitoring across all switch ports without impacting performance

Distributed DNOCs

DNOC-RTR-382&2046Cisco 6509

• best case can hold 1,000,000 ipv4 routes (or 500,000 ipv6 routes)• * 720 Gbps switch fabric capacity, up to 400Mpps switching capacity• * ethernet, SONET, serial interfaces 10Mbs-10Gps• * 40Gbs/slot capacity• * can function as a pure layer2 device or a router (or mixed)

DNOC-Rtr-2229 & 1322Foundry XMR 16000

• 16-slot high-end IPv4/IPv6/MPLS multi-service routers• Terabit-scale architecture offering up to 3.2 Tbps data capacity and

approximately 2 billion pps per system• Up to 128 10-GbE/640 1-GbE/64 OC-192/256 OC-48 ports per

system• Scalable routing via OSPF,IS-IS,BGP,OSPFv3, RIPng,PIM-DM/SM• Up to 1 million IPv4 routes in hardware to support future growth• Carrier class IPv4,IPv6,and MPLS routing featuring Foundry Direct

Routing (FDR) technology• High-capacity MPLS Layer 3/Layer 2 VPNs and IP over MPLS

routing• Enables virtual routing in non-MPLS environments via Multi-VRF• Highly resilient architecture offering redundant management

modules, switch fabrics, power supplies, cooling, and hitless failover• Flexible SONET/SDH support via either native POS interfaces or 10-

GbEWAN PHY

Internet2

SC08 Connectionsto NLR

NLR Connections

NLR Packetnet

Optiputer

Actual Topologies used by SC08 Exhibitors for demos

Caltech

Demo

Canarie

DCN Demos

OpenCloud Testbed

Questions