Network to and at CERN
description
Transcript of Network to and at CERN
![Page 1: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/1.jpg)
Network to and at CERNNetwork to and at CERN
Getting ready for LHC networking
Jean-Michel Jouanigot and Paolo Moroni
CERN/IT/CS
![Page 2: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/2.jpg)
SummarySummary
Current situationT0-T1 planning: LANT0-T1 planning: WAN
![Page 3: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/3.jpg)
21 January 2005 T0/1 network meeting 3
Current situationCurrent situation
General purpose network
Technical network
Experimental areas (pre-production)
External network
(firewall / HTAR)
![Page 4: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/4.jpg)
21 January 2005 T0/1 network meeting 4
Technical Network
CO
MPU
TE
R C
EN
TE
RR
EM
OT
E M
AJO
R ST
AR
POIN
TS
..etc..
ServerFarms
Firewall
CIXP,Internet
..etc..
General-purpose networkGeneral-purpose network
![Page 5: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/5.jpg)
21 January 2005 T0/1 network meeting 5
SR2
SR1
SR3
SR4
SR5
SR6
SR7
SR8
Technical networkTechnical network
MCR
TCR
PCR
CCR General PurposeNetwork
![Page 6: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/6.jpg)
21 January 2005 T0/1 network meeting 6
Tests + LHC pre-
production
Tests + LHC pre-
production
CIXPCIXP
GÉANT +SWITC
H
GÉANT +SWITC
H
….
….
External networkExternal network
InternetInternet
General purpose network
General purpose network
Chicago PoP
Chicago PoP
![Page 7: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/7.jpg)
21 January 2005 T0/1 network meeting 7
FirewallFirewall
This slide is intentionally left blank
![Page 8: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/8.jpg)
21 January 2005 T0/1 network meeting 8
T0-T1 planning (LAN)T0-T1 planning (LAN)
New 2.4 Tb/s backbone to interconnect LHC experiments (CERN Tier0)
general purpose network
CERN Tier1
T0-T1 WAN (regional Tier1’s)
Based on 10GE technology
Layer 3 interconnections
No central switch(es)
Redundancy via multiple 10GE paths (OSPF)
![Page 9: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/9.jpg)
21 January 2005 T0/1 network meeting 9
More about T0-T1 LANMore about T0-T1 LAN
Random paths through the backbone for load
balancing (OSPF)
IP addressing: depends on the LHC WAN implementation,
RFC1918 addresses are likely for a lot of end systems
a data mover facility can help a lot (already successfully
implemented for the BABAR experiment at IN2P3)
Default route? Maybe not necessary
Call for tender for the equipment being issued
![Page 10: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/10.jpg)
21 January 2005 T0/1 network meeting 10
..88.. ..10..
10GE->88*GE
~6000 CPU servers ..32.. ~2000 Tape and Disk servers
GPNGPN 4 LHCexperimental
areas
4 LHCexperimental
areas
T0-T1WAN
T0-T1WAN
10GE->88*GE10GE->88*GE
..88.. ..88..
….
….
10GE->32*GE 10GE->n*10GE
T0-T1 network at CERN (LAN)T0-T1 network at CERN (LAN)
RawLHCdata
External network
External network
multiple 10GE
10GE
GbE
CERN Tier1
![Page 11: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/11.jpg)
21 January 2005 T0/1 network meeting 11
Tier0 network (LHC experimental areas)Tier0 network (LHC experimental areas)
T0-T1LAN
DAQ
LHC experiment control network
GPNGPN
Low speed (management)
High speed: redundant 10GE (data)
T0-T1WAN
T0-T1WANCER
N Tier1
CERN
Tier1LHC
experiment
LHC experiment
LHC experiment
LHC experiment
LHC experiment
LHC experiment
![Page 12: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/12.jpg)
21 January 2005 T0/1 network meeting 12
T0-T1 WAN: progress
A lot of progress has been made: 10 Gb/s equipment is commonly available (although not yet
cheap): STM-64 (10GE WAN PHY), 10GE LAN
10 Gb/s capacity (SDH, wavelength, WDM over dark fibre) is
affordable
long-distance, high-speed TCP is feasible, although with
special Linux tuning
![Page 13: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/13.jpg)
21 January 2005 T0/1 network meeting 13
T0-T1 WAN: progress (continued)
More progress being made: GN2 is coming in Europe with new services and research activities
Several interesting initiatives in North America and in Europe (dark
fibre-based networks, etc.)
Several interesting monitoring tools exist or are being developed
Pre-production simulation (robust data challenge): a useful ongoing
experience
Firewall with HTAR works for non-LHC traffic and for some pre-
production
![Page 14: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/14.jpg)
21 January 2005 T0/1 network meeting 14
T0-T1 WAN: issues
Still several open questions: how will Tier1’s connect to Tier0 (directly, one upstream, layered
upstreams, …)?
backup routing ?
non-homogeneous Tier1 requirements?
any Tier1-Tier1 traffic via Tier0?
IP addressing: routable or RFC1918 ?
does every Tier1 have enough routable addresses?
and …
![Page 15: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/15.jpg)
21 January 2005 T0/1 network meeting 15
T0-T1 WAN: more issues
…what about security ?
Tier2’s ?
compatibility between GRID middleware and network design?
special tuning for WAN data transfers?
compatibility between high speed flows and some network devices
(Juniper M160)?
management, monitoring, troubleshooting?
Anything else?
![Page 16: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/16.jpg)
21 January 2005 T0/1 network meeting 16
Recommendations (I)
Allow for diverse regional requirements, but standardise
NOW on the T0-T1 physical interface: 10GE LAN PHY (LR/SR ?) STM-64/OC192 10GE WAN PHY (?) Other interfaces also possible in the pre-production phase (GbE,
multiple GbE, STM-16)
Take advantage of useful experience (robust data
challenge)
Define clearly the operational responsibilities across
multiple administrative domains
![Page 17: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/17.jpg)
21 January 2005 T0/1 network meeting 17
Recommendations (II) Select equipment which is expected to work reliably for
some years
A data mover facility (spooling system) helps with
several issues: IP addressing needs
security
WAN data transfer optimisation
Select proven and stable technology: smooth network
operations and easy troubleshooting are essential
![Page 18: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/18.jpg)
21 January 2005 T0/1 network meeting 18
Recommendations (III)
Security is essential
Monitoring is essential
Allocate suitable (routable) subnets, dedicated to
LHC production purposes
If not enough routable IP addresses, ask RIPE-NCC
for more, via the appropriate upstream LIR, and do
so NOW (or ask ARIN, or APNIC, according to the
region)
![Page 19: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/19.jpg)
21 January 2005 T0/1 network meeting 19
Recommendations (IV)
Never mind if the network is just a boring
production tool: being at the bleeding edge is
not essential in this situation
LHC physics is the research target, not LHC
networking
![Page 20: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/20.jpg)
21 January 2005 T0/1 network meeting 20
LHC WAN: a possible design
Assumptions: if … Tier1’s connect at layer 3
backup routing is a requirement and it is acceptable via research IP
networks (not more than two-three Tier1’s down at the same time)
Tier1-Tier1 traffic is allowed via Tier0 (although this would not be Tier0’s
preference…)
Tier1 and Tier0 addresses are publicly routable and every Tier1 has
allocated a SMALL number of subnets for inter-Tier0/1 traffic
BGP routing using the “natural” ASN and routable prefixes
no default route (or no default route towards T0): is it possible?
…
![Page 21: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/21.jpg)
21 January 2005 T0/1 network meeting 21
A possible design (continued)
…and if … basic security is provided via layer 3 ACLs (allowed subnets and, if
possible, port numbers)
Tier1’s may have some non-homogeneous requirements
no Tier2 directly connected to Tier0, but some may be allowed to
exchange traffic at less that 10 Gb/s
alternatively, some T0-T2 traffic may transit via an intermediate T1
a spooling system (data mover) is used as buffer between sites to
optimise long-distance data transfer and reduce public IP addresses
needs
… then …
![Page 22: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/22.jpg)
21 January 2005 T0/1 network meeting 22
LHC LANLHC LAN
Tier1Tier1
….
A possible T0-T1 WAN networkA possible T0-T1 WAN network
External network
External network
multiple 10GE
10GE or STM-64Tier1Tier1Tier1Tier1
Tier1Tier1
Tier1Tier1
Tier1Tier1
Tier1Tier1
Tier1Tier1
Tier1Tier1
10GE or multiple GbE
Data mover(spool)
Tier2
Tier2
Tier2
Tier2
![Page 23: Network to and at CERN](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814f68550346895dbd1ea7/html5/thumbnails/23.jpg)
Thank youThank you
Questions?