The size and complexity of the CERN network
-
Upload
suma-pria-tunggal -
Category
Documents
-
view
652 -
download
1
Transcript of The size and complexity of the CERN network
![Page 1: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/1.jpg)
1
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
The CERN Network
Openlab Summer 2012CERN, 6th August 2012
![Page 2: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/2.jpg)
2
Summary
- IT-CS
- CERN networks
- LHC Data Challenge
- WLCG
- LHCOPN and LHCONE
- Openlab
- Conclusions
![Page 3: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/3.jpg)
3
IT-CSCommunication systems
![Page 4: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/4.jpg)
4
IT-CS
The IT-CS group is responsible for all communication services in use at CERN for
data, voice and video
http://it-cs.web.cern.ch/it-cs/
![Page 5: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/5.jpg)
5
IT-CS organization
![Page 6: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/6.jpg)
6
Networks at CERN
![Page 7: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/7.jpg)
7
CERN accelerator complex
![Page 8: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/8.jpg)
8
High Energy Physics over IP
Most of the CERN infrastructure is controlled and managed over a
pervasive IP network
![Page 9: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/9.jpg)
9
Cryogenics
Source: http://te-dep-crg-oa.web.cern.ch/te-dep-crg-oa/te-crg-oa_fichiers/cryolhc/LHC%20Cryo_BEOP_lectures2009.pdf
27Km of pipes at -271.11° C by means of 700.000 litres of Helium: controlled over IP
![Page 10: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/10.jpg)
10
Access control
Source:https://edms.cern.ch/file/931641/1/LASS-LACS_IHM.pdf
Safety and Security: made over IP
![Page 11: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/11.jpg)
11
Remote inspections
Remote inspection of dangerous areas: robots controlled and giving feedback over WiFi and GSM IP networks
![Page 12: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/12.jpg)
12
DAQ: Data Acquisition
Source: http://aliceinfo.cern.ch/Public/Objects/Chapter2/DetectorComponents/daq_architecture.pdf
A constant stream of data from the four Detectors to disk storage
![Page 13: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/13.jpg)
13
CCC: CERN Control CentreThe neuralgic centre of the particle accelerator: over IP
![Page 14: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/14.jpg)
14
CERN data network
- 150 routers- 2200 Switches- 50000 connected devices- 5000km of optical fibres
![Page 15: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/15.jpg)
15
Network Provisioning and Management System
- 250 Database tables
- 100,000 Registered devices
- 50,000 hits/day on web user interface
- 1,000,000 lines of codes
- 11 years of development
![Page 16: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/16.jpg)
16
Monitoring and OperationsThe whole network is monitored and operated by the CERN NOC (Network Operation Centre)
![Page 17: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/17.jpg)
17
IPv6
IPv6 dual stack network deployment on going: ready in 2013
Already available: dual-stack testbed
More information: http://cern.ch/ipv6
almost
![Page 18: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/18.jpg)
18
LHC Data Challenge
![Page 19: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/19.jpg)
19
Collisions in the LHC
![Page 20: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/20.jpg)
20
Comparing theory...
Simulated production of a Higgs event in ATLAS
![Page 21: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/21.jpg)
21
.. to real events
Higgs event in CMS
![Page 22: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/22.jpg)
22
Data flow
4 Experiments
3 PBytes/s
2 GBytes/sto the CERN computer center
Store on disk and tape
World-Wide Analysis
Export copies
Create sub-samples
col2f
2f
3Z
ff2Z
ffee2Z
0
ff
2z
2Z
222Z
2Z0
ffff
N)av(26
m and
m
12
withm/)m-(
_
__
×+×=ΓΓ
ΓΓ=
Γ+Γ×≈
ππσ
σσ
FG
ss
s
PhysicsExplanation of nature
10 GBytes/s 4 GBytes/s
1 TByte/s ?Distributed + local
Filter and first selection
![Page 23: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/23.jpg)
23
Data Challenge
- 40 million collisions per second
- After filtering, 100 collisions of interest persecond
- 1010 collisions recorded each year = 15 Petabytes/year of data
![Page 24: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/24.jpg)
24
Computing model
![Page 25: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/25.jpg)
25
Last months data transfers
![Page 26: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/26.jpg)
26
WLCGWorldwide LHC Computing Grid
![Page 27: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/27.jpg)
27
WLCG
Distributed Computing Infrastructure for LHC experiments
Collaborative effort of the HEP community
![Page 28: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/28.jpg)
28
WLCG resources
WLCG sites:- 1 Tier0 (CERN)
- 11 Tier1s
- ~140 Tier2s
- >300 Tier3s worldwide
- ~250,000 CPUs
- ~ 150PB of disk space
![Page 29: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/29.jpg)
29
CERN Tier0 resources
March 2012
Disks 64000
Raw Disk Capacity (TB) 63000
Memory Modules 56000
RAID controllers 3750
Servers 11000
Processors 15000
Cores 64000
HEPspec06 480000
Tape drives 160
Tape cartridges 45000
Tape slots 56000
Tape capacity(TB) 34000
High Speed routers 23
Ethernet switches 500
10Gbps ports 3000
100Gbps ports 48
![Page 30: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/30.jpg)
30
CERN Tier0 LCG network
Border routers
Distribution routers
LCG access switches
Servers
Access switches
Core routers
... x892 (max)
170G aggregated
100G links
1G or 10G links
40G links
10G or 40G links
CERN Campus
LHC Experiments
Tier2/3s Tier1s
![Page 31: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/31.jpg)
31
Trends
Virtualization mobility (Software Defined Networks)
Commodity Servers with 10G NICs
High-end Servers with 40G NICs
40G and 100G interfaces on switches and routers
![Page 32: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/32.jpg)
32
LHCOPNLHC Optical Private Network
![Page 33: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/33.jpg)
33
Tier0-Tier1s network
![Page 34: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/34.jpg)
34
A collaborative effort
Designed, built and operated by the Tier0-Tier1s community
Links provided by the Research and Education network providers: Geant, USLHCnet, Esnet, Canarie, ASnet, Nordunet, Surfnet, GARR, Renater, JANET.UK, Rediris, DFN, SWITCH
![Page 35: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/35.jpg)
35
Technology
- Single and bundled long distance 10G ethernet links
- Multiple redundant paths. Star+PartialMesh topology
- BGP routing: communities for traffic engineering, load balancing.
- Security: only declared IP prefixes can exchange traffic.
![Page 36: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/36.jpg)
36
Traffic to the Tier1s
![Page 37: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/37.jpg)
37
Monitoring
![Page 38: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/38.jpg)
38
LHCONELHC Open Network Environment
![Page 39: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/39.jpg)
39
Driving the change
“The Network infrastructure is the most reliable service we have”
“Network Bandwidth (rather than disk) will need to scale more with users and
data volume”
“Data placement will be driven by demand for analysis and not pre-
placement”
Ian Bird, WLCG project leader
![Page 40: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/40.jpg)
40
Change of computing model (ATLAS)
![Page 41: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/41.jpg)
41
New computing model
- Better and more dynamic use of storage
- Reduce the load on the Tier1s for data serving
- Increase the speed to populate analysis facilities
Needs for a faster, predictable, pervasive network connecting Tier1s and Tier2s
![Page 42: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/42.jpg)
42
Requirements from the Experiments
- Connecting any pair of sites, regardless of the continent they reside
- Bandwidth ranging from 1Gbps (Minimal), 5Gbps (Nominal), 10G and above (Leadership)
- Scalability: sites are expected to grow
- Flexibility: sites may join and leave at any time
- Predictable cost: well defined cost, and not too high
![Page 43: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/43.jpg)
43
Needs for a better network
- more bandwidth by federating (existing) resources
- sharing cost of expensive resources- accessible to any TierX site
=
LHC Open Network Environment
![Page 44: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/44.jpg)
44
LHCONE concepts
- Serves any LHC sites according to their needs and allowing them to grow
- A collaborative effort among Research & Education Network Providers
- Based on Open Exchange Points: easy to join, neutral
- Multiple services: one cannot fit all
- Traffic separation: no clash with other data transfer, resource allocated for and funded by HEP community
![Page 45: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/45.jpg)
45
LHCONE architecture
![Page 46: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/46.jpg)
46
LHCONE building blocks
- Single node Exchange Points- Continental/regional Distributed Exchange Points
- Interconnect circuits between Exchange Points
These exchange points and the links in between collectively provide LHCONE services and operate under a
common LHCONE policy
![Page 47: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/47.jpg)
47
The underlying infrastructure
![Page 48: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/48.jpg)
48
LHCONE services
- Layer3 VPN
- Point-to-Point links
- Monitoring
![Page 49: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/49.jpg)
49
Openlab and IT-CS
![Page 50: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/50.jpg)
50
Openlab project:
CINBAD
![Page 51: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/51.jpg)
51
CINBAD
CERN Investigation of Network Behaviour and Anomaly Detection
Project Goals:Understand the behaviour of large computer networks (10’000+ nodes) in High Performance Computing or large Campus installations to be able to:
● detect traffic anomalies in the system● perform trend analysis● automatically take counter measures ● provide post-mortem analysis facilities
Resources:- In collaboration with HP Networking- Two Engineers in IT-CS
![Page 52: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/52.jpg)
52
Results
Project completed in 2010
For CERN:Designed and deployed a complete framework (hardware and software) to detect anomalies in the Campus Network (GPN)
For HP:Intellectual properties of new technologies used in commercial products
![Page 53: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/53.jpg)
53
CINBAD Architecture
data sources
collectors
storage
analysis
![Page 54: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/54.jpg)
54
Openlab project:
WIND
![Page 55: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/55.jpg)
55
WIND
Wireless Infrastructure Network Deployment
Project Goals- Analyze the problems of large scale wireless deployments and
understand the constraint- Simulate behaviour of WLAN- Develop new optimisation algorithms
Resources:- In collaboration with HP Networking- Two Engineers in IT-CS- Started in 2010
![Page 56: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/56.jpg)
56
Needs
Wireless LAN (WLAN) deployments are problematic:
● Radio propagation is very difficult to predict
● Interference is an ever present danger
● WLANs are difficult to properly deploy
● Monitoring was not an issue when the first standards were developed
● When administrators are struggling just to operate the WLAN, performance optimisation is often forgotten
![Page 57: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/57.jpg)
57
Example: Radio interferences
Max data rate in 0031-S: The APs work on 3 independent channels
Max data rate in 0031-S: The APs work on the same channel
![Page 58: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/58.jpg)
58
Expected results
Extend monitoring and analysis tools
Act on the network- smart load balancing- isolating misbehaving clients- intelligent minimum data rates
More accurate troubleshooting
Streamline WLAN design
![Page 59: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/59.jpg)
59
Openlab project:
ViSION
![Page 60: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/60.jpg)
60
ViSION
Project Goals: - Develop a SDN traffic orchestrator using OpenFlow
Resources:- In collaboration with HP Networking- Two Engineers in IT-CS- Started in 2012
![Page 61: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/61.jpg)
61
GoalsSDN traffic orchestrator using OpenFlow:
● distribute traffic over a set of network resources● perform classification (different types of applications and
resources)● perform load sharing (similar resources).
Benefits:● improved scalability and control than traditional networking
technologies
![Page 62: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/62.jpg)
62
From traditional networks...
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
App App App
Specialized Packet Forwarding Hardware
OperatingSystem
OperatingSystem
OperatingSystem
OperatingSystem
OperatingSystem
App App App
Closed boxes, fully distributed protocols
![Page 63: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/63.jpg)
63
.. to Software Defined Networks (SDN)
App
Simple Packet Forwarding Hardware
Simple Packet Forwarding Hardware
Simple Packet Forwarding Hardware
App App
Simple Packet Forwarding Hardware
Simple Packet Forwarding Hardware
Network Operating System
1. Open interface to hardware (OpenFlow)
3. Well-defined open API2. At least one good operating system
Extensible, possibly open-source
![Page 64: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/64.jpg)
64
OpenFlow exampleController
PC
HardwareLayer
SoftwareLayer
Flow Table
MACsrc
MACdst
IPSrc
IPDst
TCPsport
TCPdport
Action
OpenFlow Client
port 4port 3port 2port 1
**5.6.7.8*** port 1
Hardware Forwarding table remotely controlled
Hardware Forwarding table remotely controlled
![Page 65: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/65.jpg)
65
Conclusions
![Page 66: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/66.jpg)
66
Conclusions
- The Data Network is an essential component of the LHC instrument
- The Data Network is a key part of the LHC data processing and will become even more important
- More and more security and design challenges to come
![Page 67: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/67.jpg)
67
Credits
Artur Barczyk (LHCONE)
Dan Savu (VISION)
Milosz Hulboj (WIND and CINBAD)
Ryszrard Jurga (CINBAD)
Sebastien Ceuterickx (WIND)
Stefan Stancu (VISION)
Vlad Lapadatescu (WIND)
![Page 68: The size and complexity of the CERN network](https://reader033.fdocuments.us/reader033/viewer/2022042715/558cdadad8b42acb118b461a/html5/thumbnails/68.jpg)
68
What's next
SWAN: Space Wide Area Network :-)