The Worldwide LHC Computing Grid
14
The Worldwide LHC Computing Grid Frédéric Hemmer IT Department Head Visit of INTEL ISEF CERN Special Award Winners 2012 Thursday, 21 st June 2012
description
The Worldwide LHC Computing Grid. Visit of INTEL ISEF CERN Special Award Winners 2012 Thursday, 21 st June 2012. Frédéric Hemmer IT Department Head. The LHC Data Challenge. The accelerator will run for 20 years - PowerPoint PPT Presentation
Transcript of The Worldwide LHC Computing Grid
The Worldwide LHC Computing Grid
Frédéric HemmerIT Department Head
Visit of INTEL ISEF
CERN Special Award Winners 2012
Thursday, 21st June 2012
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/it
2
The LHC Data Challenge
• The accelerator will run for 20 years
• Experiments are producing about 20 Million Gigabytes of data each year (about 3 million DVDs – 700 years of movies!)
• LHC data analysis requires a computing power equivalent to ~100,000 of today's fastest PC processors
• Requires many cooperating computer centres, as CERN can only provide ~20% of the capacity
June 2012 - Frédéric Hemmer
• A distributed computing infrastructure to provide the production and analysis environments for the LHC experiments
• Managed and operated by a worldwide collaboration between the experiments and the participating computer centres
• The resources are distributed – for funding and sociological reasons
• Our task was to make use of the resources available to us – no matter where they are located
June 2012 - Frédéric Hemmer
WLCG – what and why?
Tier-0 (CERN):• Data recording• Initial data reconstruction• Data distribution
Tier-1 (11 centres):• Permanent storage• Re-processing• Analysis
Tier-2 (~130 centres):• Simulation• End-user analysis
3
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/it
June 2012 - Frédéric Hemmer 4
Global Lambda Integrated Facility
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/it
June 2012 - Frédéric Hemmer 5
Data acquired in 2012
2012 Data written: Total 9.4 PB to end May>3 PB in May (cf 2 PB/month in 2011)
Data accessed from tape, 2012
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/it
June 2012 - Frédéric Hemmer 6
Data transfersGlobal transfers > 10 GB/s (1 day)
Global transfers (last month)
CERN Tier 1s (last 2 weeks)
WLCG – no stop for computingActivity on 3rd Jan
Problem - Technology Explosion with NGS
Feb
08
Apr
08
Jun
08
Aug
08
Oct
08
Dec
08
Feb
09
Apr
09
Jun
09
Aug
09
Oct
09
Dec
09
Feb
10
Apr
10
June
10
Aug
10
Oct
10
Dec
10
Feb
11
Apr
il 11
June
11
Aug
ust 1
1
0
5,000,000,000
10,000,000,000
15,000,000,000
20,000,000,000
25,000,000,000
30,000,000,000
35,000,000,000
Bases Sequenced / Sample / Run @ EMBL (Illumina)
8
Sequence Production & IT Infrastructure at EMBL
Compute Power:2000+ CPU Cores, 6+ TB RAM
Storage:1+ PB High Performance Disk
4 x Ilumina HiSeq2000
25 TB data each week
2 x Ilumina GAIIx
10
NGS - The Big Picture
• ~ 8.7 million species in the world (estimate)• ~ 7 billion people• Sequencers exist in both large centres & small research
groups• > 200 Ilumina HiSeq sequencers in Europe alone
=> capacity to sequence 1600 human genomes / month• Largest centre: Beijing Genomics Institute (BGI)
• 167 sequencers, 130 HiSeq• 2,000 human genomes / day
• 500-1000 Hiseq devices worldwide today• 3-6 PB /day• 1.1 – 2.2 Exabytes / year
11
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/it
June 2012 - Frédéric Hemmer 13
The CERN Data Centre in Numbers
• Data Centre Operations (Tier 0)– 24x7 operator support and System Administration services to support 24x7
operation of all IT services.– Hardware installation & retirement
• ~7,000 hardware movements/year; ~1800 disk failures/year
– Management and Automation framework for large scale Linux clusters
Xeon 51502%
Xeon 516010%
Xeon E5335
7%Xeon
E534514%
Xeon E5405
6%
Xeon E541016%
Xeon L5420
8%
Xeon L552033%
Xeon 3GHz4%
Fujitsu3%
Hitachi23% HP
0% Maxtor
0% Seagate15%
Western Digital
59%
Other0%
High Speed Routers(640 Mbps → 2.4 Tbps)
24
Ethernet Switches 350
10 Gbps ports 2000
Switching Capacity 4.8 Tbps
1 Gbps ports 16,939
10 Gbps ports 558
Racks 828
Servers 8938
Processors 15,694
Cores 64,238
HEPSpec06 482,507
Disks 64,109
Raw disk capacity (TiB) 63,289
Memory modules 56,014
Memory capacity (TiB) 158
RAID controllers 3,749
Tape Drives 160
Tape Cartridges 45000
Tape slots 56000
Tape Capacity (TiB) 34000
IT Power Consumption 2456 KW
Total Power Consumption 3890 KW
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/it
June 2012 - Frédéric Hemmer 14
Scaling CERN Data Center(s) to anticipated needsCERN Data Center dates back to the 70’s
– Now optimizing the current facility (cooling automation, temperatures, infrastructure)
Renovation of the “barn” for accommodating 450 KW of “critical” IT loads – an EN, FP, GS, HSE, IT joint venture
Exploitation of 100 KW of remote facility down town– Understanding costs, remote dynamic
management, ensure business continuity
Exploitation of a remote Data center in Hungary– 100 Gbps connections– Agile infrastructure
– virtualization
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/it
15June 2012 - Frédéric Hemmer
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/it
16June 2012 - Frédéric Hemmer
