Network Architecture and IPv6 Deployment at CERN
description
Transcript of Network Architecture and IPv6 Deployment at CERN
![Page 1: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/1.jpg)
ITDepartment
Network Architecture and IPv6 Deployment at CERN
CHEP Oct [email protected]
Co-authors: [email protected], [email protected] IT/Communication Systems
![Page 2: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/2.jpg)
ITDepartment2
Agenda
1. Network Infrastructures
2. LCG Architecture
3. Management
4. IPv6 Deployment Status
![Page 3: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/3.jpg)
ITDepartment3
1. Network Infrastructure
![Page 4: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/4.jpg)
ITDepartment4
• Interconnects all infrastructures• Extends between Geneva-Wigner • Non-blocking 1Tbps• IPv6 Ready• OSPF backbone• Security policies• Problem isolation
CORE Network
CORE
Technical
GeneralPurpose
External
LHCComputingGrid
Experiments
![Page 5: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/5.jpg)
ITDepartment5
• Technical services– Safety, electricity, cooling,…
• LHC Operation– Cryogenics, vacuum,…
• Industrial systems
Technical Network
Devices 8,073Switches 485HP Routers 24IPv4/IPv6 Dual Stack NO
CORETechnical
GeneralPurpose External
LHCComputingGrid
Experiments
![Page 6: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/6.jpg)
ITDepartment6
• IT/CS provides– Detector Control Network
• ALICE• ATLAS• CMS
– Data Recording to T0• 20Gbps, up to 50Gbps• 25 PB/year
Experiments
Devices 6,111Switches 390HP Routers 15IPv4/IPv6 Dual Stack NO
CORETechnical
GeneralPurpose External
LHCComputingGrid
Experiments
![Page 7: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/7.jpg)
ITDepartment7
• High Performance Network• 5.28 Tbps Non-blocking Switching Fabric• WLCG Tier0
– Long-term storage– Distribution WLCG– 1.5 PB/day
LHC Computing Grid
Devices 8,902Switches 588Brocade Routers 13IPv4/IPv6 Dual Stack YES
CORETechnical
GeneralPurpose External
LHCComputingGrid
Experiments
![Page 8: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/8.jpg)
ITDepartment8
• Public general purpose connections– Full BGP Internet routing table– Geant, CIXP, ISPs
• Private WLCG– LHCOPN
• 70Gbps peaks to T1– LHCONE
External Network
Brocade Routers 8BGP Peerings 86Aggregated BW 232 GbpsIPv4/IPv6 Dual Stack YES
CORETechnical
GeneralPurpose External
LHCComputingGrid
Experiments
![Page 9: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/9.jpg)
ITDepartment9
• Desktop computing• Wired and wireless• Central Services
– AFS, www, mail, databases
General Purpose Network
Users 14,592Buildings 650Devices 114,061Switches 1,550WiFi Access Points 1,514HP Routers 100Brocade Routers 7IPv4/IPv6 Dual Stack YES*
CORETechnical
GeneralPurpose External
LHCComputingGrid
Experiments
![Page 10: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/10.jpg)
ITDepartment10
2. LCG Network Architecture
GENEVA WIGNER
Internet
CORE
![Page 11: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/11.jpg)
ITDepartment11
• Network Access modeled as Service• A Service is a broadcast domain
– Providing access to end systems– IPv6/IPv4 Dual Stack
• Features are defined by a Service Type– Jumbo, LACP, tagged, loop protect, access-list
protect, OSPF, VRRP, …• A ToR Switch provides a Service• A Router interconnects multiple Services
Building block: Service
![Page 12: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/12.jpg)
ITDepartment12
Service Bandwidth• Service capacity depends on Service purpose• Blocking Factor: 2 for CPUs, 5 for Storage
x 1Gbps
m x 10 Gbps
10 Gbps x
n x 10 Gbps
CPUs rack Storage rackDistribution Router
![Page 13: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/13.jpg)
ITDepartment13
Rows of Racks Distribution RoutersBackbone Routers
100s of 10 Gbps
100s of 10 Gbps
100s of 10 Gbps
100 Gbps links
Scaling the Data Center
![Page 14: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/14.jpg)
ITDepartment14
Backbone
LCG Border Routers
100 Gbps links
∑ 140 Gbps
∑ 20 Gbps
CORENetwork
∑ 60 Gbps
Internet∑ 12 Gbps
ExternalNetwork
Racks Distribution
Worldwide LCG
![Page 15: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/15.jpg)
ITDepartment15
CORENetwork
Internet
∑ 240 GbpsRacks Distribution
Backbone
Extending the Tier0 to Wigner
![Page 16: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/16.jpg)
ITDepartment16
Internet
CORE
Backbone Routers
Racks Distribution RacksDistribution
MPLS
GENEVA WIGNERBackbone Backbone
WLCG Tier0
![Page 17: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/17.jpg)
ITDepartment17
3. Network Management
Public Web
SOAP
Admin Web
Configuration Manager
Monitoring
![Page 18: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/18.jpg)
ITDepartment18
Racks Distribution
Dyn-A4 three columns example of a distribution router configuration
I think I can fix it!
![Page 19: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/19.jpg)
ITDepartment19
Network Database
• Buildings and cabling infrastructure• IPv4/IPv6 addressing schemas• Services and service features• Devices (Servers, VMs, desktops, smartphones, …)• Switches• Routers/Firewalls• Security for the Infrastructure (firewall rules)
![Page 20: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/20.jpg)
ITDepartment20
Interfaces to the Network Database
Public Web
SOAP
Admin Web
Configuration Manager
Monitoring
Net AdminsFirst Line
Sys Admins
Users
Tech Admins
![Page 21: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/21.jpg)
ITDepartment21
Software-based Network configuration
Configuration Manager
Net AdminsFirst Line
Network Services• DNS• DHCP• RADIUS
Automatic Updates
CORETechnical
GeneralPurpose External
LHCComputingGrid
Experiments
![Page 22: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/22.jpg)
ITDepartment22
4. IPv6 Deployment Status
![Page 23: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/23.jpg)
ITDepartment23
2012
Network Database: Schema and Data IPv6 Ready
Configuration Manager supports IPv6 routing
Admin Web: IPv6 integrated
2013
The Data Center is Dual-Stack
Gradual deployment on the routing infrastructure starts
NTPv6 and DNSv6
TodayDHCPv6
• Infrastructure is Dual-Stack
• Firewallv6 automated configuration
• User Web and SOAP integrate IPv6
• Automatic DNS AAAA configuration
![Page 24: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/24.jpg)
ITDepartment24
• Identical performance, common tools and services• Dual Stack, dual routing
– OSPFv2/OSPFv3– BGP ipv4 and ipv6 peers
• Service managers decide when ready for IPv6• Devices must be registered
– SLAAC disabled– RAs: Default Gateway + IPv6 prefixes no-autoconfig– DHCPv6
• MAC addresses as DUIDs: painful without RFC6939• ISC has helped a lot (βcode implementing classes for ipv6)• DHCPv6 clients might not work ‘out of the box’
IPv4 / IPv6 same portfolio
![Page 25: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/25.jpg)
ITDepartment25
• The Network is ready to accommodate the new demands after Long Shutdown 1
• Before Eo2013, IPv6 will be fully deployed and available to the CERN community
Conclusions
![Page 26: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/26.jpg)
ITDepartment26
Thank you!
![Page 27: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/27.jpg)
ITDepartment27
Extra Slides
![Page 28: Network Architecture and IPv6 Deployment at CERN](https://reader036.fdocuments.us/reader036/viewer/2022062323/568163fe550346895dd59a25/html5/thumbnails/28.jpg)
ITDepartment28
CERN
Area ~600,000m2
Buildings 646
Staff and Users 14,592
Devices Registered 170,475
Data Centers Geneva Wigner 2013Power 3,500KW ~900KW
Racks 828 90
Servers 10,173 ~1,200
Routers 22 6
100Gbps ports 60 18
ToR Switches 662 140
ToR Switching
1Gbps ports 22,776 3,072
10Gbps ports 4,284 528
Storage
Disks 79,505
Raw disk capacity (TiB) 124,660
Tape Drives 160
Data on Tape (PiB) 65
L2 Switching
Switches 2726
1 Gbps ports 91230
10 Gbps ports 5656
L3 Switching
Routers 161
1 Gbps ports 5976
10 Gbps ports 2248
100 Gbps ports 78
WiFi
Access Points 1,514
Devices seen/day ~7,000