NCCS Network Roadmap
10
Presented by NCCS Network Roadmap Josh Lothian High Performance Computing Operations National Center for Computational Sciences
description
NCCS Network Roadmap. Josh Lothian High Performance Computing Operations National Center for Computational Sciences. NCCS network roadmap. Summary: Hybrid Ethernet/Infiniband (IB) network to provide both high-speed wide-area connectivity and uber-speed local-area data movement. 2007 - PowerPoint PPT Presentation
Transcript of NCCS Network Roadmap
Presented by
NCCS Network Roadmap
Josh LothianHigh Performance Computing Operations
National Center for Computational Sciences
2 Lothian_Networking_SC07
NCCS network roadmap
2007 Scaled Ethernet to meet wide-area needs and current day local-
area data movement Developed wire-speed, low-latency perimeter security to fully
utilize 10Gb production and research WAN connections Began IB LAN deployment
2008 Continue building out IB LAN infrastructure to satisfy the file
system needs of the Baker system Test Lustre on IB/WAN for possible deployment Increase security on 10Gb WAN connections
Summary: Hybrid Ethernet/Infiniband (IB) network to provide both high-speed wide-area connectivity and uber-speed local-area data movement
3 Lothian_Networking_SC07
Ethernet[O(10Gb/s)]
Ethernet core scaled to match wide-area
connectivity and archive
Infiniband core scaled to match central file system
and data transfer
NCCS network roadmap summary
Gateway
Viz
High-PerformanceStorage System
(HPSS)
Jaguar
Lustre
Baker
Infiniband[O(100 Gb/s)]
4 Lothian_Networking_SC07
Juniper T640
Ciena CoreStream
Juniper T320
Force10 E600
Cisco 6509NSTG
TeraGrid
CCS
DOE UltraScience/NSF CHEETAH
CCS
OC-192 to TeraGrid
2 x OC-192 to USN
OC-192 to CHEETAH
NCCS WAN overview
OC-192 to Internet 2
OC-192 to Esnet
Internal 10Gb Link
Internal 1Gb Link
Ciena CoreDirector
5 Lothian_Networking_SC07
4
4
4
20
32
48
4848
48
4848
48
48
48
48
20 DDR
Infiniband
6-12 DDR
64 DDR
24 SDR
87 SDR
6-12 DDR
24-48 SDR
32 DDR
32
Ethernet
32 DDR
Spider30
Spider10
Viz
HPSS
Devel
E2E
Jaguar
NCCS Network 2007
Force10 E1200
Voltaire ISR2012
Cisco 7024D
Voltaire ISR9288
6 Lothian_Networking_SC07
50 DDR
24 SDR
20 DDR
300 DDR(50 DDR/link)
64 DDR
32 DDR
87 SDRJaguar
Viz
HPSS
E2E
Devel
Spider240Baker
50 DDR
48-96 SDR(16-32 SDR/link)
NCCS IB network 2008
Spider10Spider10
Spider30Spider30
20 DDR
32 DDR
Voltaire ISR2012
Voltaire ISR9288
Cisco 7024D
7 Lothian_Networking_SC07
2007 milestones
April 2007—Completed 10 G WAN upgrade
June 2007—Completed VL/QOS testing
November 2007—Complete fiber IB cable testing
December 2007—IBoIP/SDP decision point
8 Lothian_Networking_SC07
2008 milestones
January—4x QDR testing
February—Deploy flow-based perimeter security
February—Complete 4X QDR testing
March—Enable routing of XT3 HSN to production network
June—Deploy redundant links between IB switches
9 Lothian_Networking_SC07
IB over WAN testing
Placed 2 x Obsidiam Longbow devices between Voltaire 9024 and Voltaire 9288
Provisioned loopback circuits of various lengths on the DOE UltraScience Network and ran test
RDMA Test Results:Local: 7.5 Gbps (Longbow to Longbow)
ORNL ↔ ORNL (0.2 mile): 7.5 Gbps
ORNL ↔ Chicago (1400 miles): 7.46 Gbps
ORNL ↔ Seattle (6600 miles): 7.23 Gbps
ORNL ↔ Sunnyvale (8600 miles): 7.2 Gbps
20 81
ObsidianLongbow
CienaCD-CI(SNV)
4x Infiniband SDR
OC-192 SONET
Lustre End-to-End
CienaCD-CI
(ORNL)
ObsidianLongbow
Voltaire9024
Voltaire9288
DOE UltraScience
Network
10 Lothian_Networking_SC07
Contact
Josh LothianHigh Performance Computing OperationsNational Center for Computational Sciences(865) [email protected]
10 Lothian_Networking_SC07
