Swiss GRID Activities Christoph Grab Lausanne, March 2009 Lausanne, March 2009.

Swiss GRID Activities Swiss GRID Activities

Christoph Grab Lausanne, March 2009Lausanne, March 2009

http://www.cscs.ch/

Christoph Grab, ETH 2

Global Grid CommunityGlobal Grid Community

The global GRID community …





Grids in Switzerland (Just Some)Grids in Switzerland (Just Some)Grids in Switzerland (Just Some)Grids in Switzerland (Just Some)

Concentrate on HEP…





WLCG - Hierarchy Model …WLCG - Hierarchy Model …WLCG - Hierarchy Model …WLCG - Hierarchy Model …

Swiss Tier-2 @CSCS

LHCb Tier-3

ATLAS Tier -3

CMS Tier-3

Status of the

Swiss Tier-2

Regional Centre


Swiss Tier-2: Facts and Figures (1)Swiss Tier-2: Facts and Figures (1)

The Swiss Tier-2 is operated by a collaboration of CHIPP and CSCS(Swiss Centre of Scientific Computing of ETHZ), located in Manno (TI).

Properties :

One Tier-2 for all three expts. CMS, ATLAS + LHCb; provides:

Simulation for experiment’s community (supply WLCG pledges) End-user analysis for Swiss community support (operation and data supply) for Swiss Tier-3 centres

Standard LINUX compute cluster “PHOENIX” (similar to other Tier-2s)

Hardware setup increased incrementally in phases :

Technology choice so far: SUN blade centres + quad Opterons.

Final size to be reached by ~ 2009/2010:

NO tapes


Swiss Tier-2 : Cluster EvolutionSwiss Tier-2 : Cluster Evolution

Growth corresponds to Swiss commitment in terms of compute resources supplied to the expt’s according to the signed MoU with WLCG.

In operation now: 960 cores 1600 kSI2k; total 520 TB storage

Last phase planned for Q4/09: 2500 kSI2k; ~ 1 PB storage

0

500

1000

1500

2000

2500

3000

Q2/04 Q2/05 Q4/06 Q1/08 Q4/08 Q4/09

Swiss Tier-2 Size Evolution

Va

lue

s

CPU [kSI2k] Storage [TB]

Phase A

Phase BOperational

Phase CPlanned Q4/09

Phase-0

CG_0209

Christoph Grab May, 2008 8

Swiss LHC Tier-2 cluster Swiss LHC Tier-2 cluster “PHOENIX”“PHOENIX” Swiss LHC Tier-2 cluster Swiss LHC Tier-2 cluster “PHOENIX”“PHOENIX”

System Phase B operational since Nov 2008System Phase B operational since Nov 2008

CPUs: total of 1600 SI2KSUN SB8000P blade centres; AMD Opterons 2.6 GHz CPU (quad)

Storage: 27 X4500 - systems net capacity of 510 TB


Incremental CPU-usageover last 4 years 4.5 106 hours= 512 CPU-years

Swiss Tier-2 usage Swiss Tier-2 usage (6/05-2/09)(6/05-2/09)

Tier-2 is up and has been in stable operation ~4 years ! continuous contributions of resources to experiments.

Our Tier-2 size is in line with other Tier-2 (e.g. London T2)

Phase A CMS

ATLAS

Phase B

LHCb

CSCS-LCG2


Normalised CPU time per month Normalised CPU time per month (3/07-2/09)(3/07-2/09)

Shares between VOs varies over time (production, challenges…)

Spare cycles given to other VO (eg. H1, theory (CERN) ... )

CMS ATLAS

LHCbH1others

CSCS-LCG2


Shares of normalised CPU per VO Shares of normalised CPU per VO (3/05-2/09)(3/05-2/09)

Shares between VOs overall reasonably balanced.

Reliability

CMS ATLAS

LHCbH1others

CSCS-LCG2



Manpower for Tier-2:

Operation at CSCS: ~ 2.5 FTEs (IT experts, about 5 persons)

support of experiment specifics by scientists of experiments;one contact person per experiment in total ~2 FTE.

Financing :

Financing of hardware mainly through SNF/FORCE (~90%), with some contributions by Universities + ETH + PSI;

Operations and infrastructure provided by CSCS (ETHZ),additional support physicists provided by institutes.

Network traffic:

routing via SWITCH : two redundant lines to CERN and Europe

transfer rates reached up to 10 TB /day from FZK (and CERN)

FZK (Karlsruhe) is the associated Tier-1 for Switzerland.


Financing : Hardware and service, no manpower :

R&D financed by institutes in 2004

Total financial contributions (2004-2008) for incremental setup of Tier-2 cluster hardware only:

by Universities + ETH + EPFL + PSI ~200 kCHF

by Federal funding (FORCE/SNF) 2.4 MCHF

Planned investments:

in 09 : last phase ~ 1.3 MCHF.

>=2010 onwards: rolling replacements ~ 700 kCHF/year

Total investment up to Q1/2009 of ~ 2.6 MCHF; annual recurring costs expected (>2009) ~ 700 kCHF




Manpower for Tier-2:

Operation at CSCS: ~ 2.5 FTEs (IT experts, about 5 persons)

support of experiment specifics by scientists of experiments;one contact person per experiment in total ~2 FTE.

Financing (HW and service, no manpower) :

Financing of hardware mainly through SNF/FORCE (~90%), with some contributions by Universities + ETH + PSI;

Operations and infrastructure provided by CSCS (ETHZ)

Network traffic:

routing via SWITCH : two redundant lines from CSCS to CERN and Europe(SWITCH = Swiss academic network provider)

FZK (Karlsruhe) is the associated Tier-1 for Switzerland.

transfer rates reached up to 10 TB /day from FZK (and CERN)


Swiss Network TopologySwiss Network Topology

SWITCHlan Backbone: Dark Fiber Topology October 2008

10 Gbps

10 Gbps

T0 at CERNT0 at CERN T2 at CSCST2 at CSCS

T1 at FZKT1 at FZK


Status of the

Swiss Tier-3

Centres


Swiss Tier-3 EffortsSwiss Tier-3 Efforts

ATLAS : operates the Swiss ATLAS Grid federation of clusters at Bern uses local HEP cluster + shares university resources Geneva operates local cluster +T2

CMS : ETHZ + PSI+ UZH run a combined Tier-3

located at and operated by PSI IT

LHCb : EPFL : operates new large local cluster UZH uses local HEP + shares university resources

Large progress seen over last year for all 3 experiments.

Close national collaboration between Tiers:

Tier-3 contacts are ALSO experiment’s site contacts for CH Tier-2.

close contacts to Tier-1 at FZK .


Swiss Network and Tiers LandscapeSwiss Network and Tiers Landscape


10 Gbps

10 Gbps


T1 at FZKT1 at FZK

CMS Tier-3CMS Tier-3

ATLAS ATLAS Tier-3 GeTier-3 Ge

ATLAS ATLAS Tier-3 BeTier-3 Be

LHCb LHCb Tier-3 EPFLTier-3 EPFL

LHCb LHCb Tier-3 UZHTier-3 UZH


Summary: Swiss Tier-3 Efforts Summary: Swiss Tier-3 Efforts (Q1/09)(Q1/09)

Nr cores CPU(kSI2k)

Storage (TB)

Comments

ATLAS BE GE

30+300s188

~600 320

3044

BE: GRID usage since 2005 for Atlas productionGE: identical SW-environment to CERN; direct line to CERN.

CMS ETHZ, PSI, UZH

72 ~250 105Operates GRID storage element and user interface to enable direct GRID access.

LHCb EPFL UZH

464shared

~800125

3615

EPFL is a DIRAC pilot site identical machines as in pit

UZH:MC production; shared

Total Tier-3 ~2100 230 cf: Tier-2: 1600 kSI2k, 520 TB

Tier-3 capacities : similar size in CPU as Tier-2 ; and ~ 50% diskSubstantial investment of resources for MC production+local analysis.(… more details in backup slides)

Note: CPU numbers are estimates; upgrades in progress …

Status 1.3.09


Swiss non-LHC GRID Efforts (1)Swiss non-LHC GRID Efforts (1)

Physics community: Theory : some use idle Tier-2 / Tier-3 resources (CERN,…) HEP Neutrino community: own clusters, or CERN lxplus… PSI : synergies of Tier-3 know-how for others: ESRFUP project

(collab. with ESRF, DESY and SOLEIL for GRID @synchrotrons) Others use their own resources (smaller clusters ..)

Several other Grid projects exist in the Swiss academic sector:

EU Projects: EGEE-II, KnowARC, DILIGENT, CoreGrid, GridChem ...International Projects: WLCG, NorduGRID, SEPAC, PRAGMA, …National Projects: Swiss Bio Grid, G3@UZH, AAA/Switch ..Various Local Grid activities (infrastructure, development…): Condor CampusGrids, local University clusters …


Swiss non-LHC GRID Efforts (2)Swiss non-LHC GRID Efforts (2)Swiss non-LHC GRID Efforts (2)Swiss non-LHC GRID Efforts (2)

SWING: Swiss national GRID association:by ETHZ, EPFL, Cantonal Univ., Univ. of Applied Sciences, and by CSCS, SWITCH, PSI, ...

Provide a platform for interdisciplinary collaboration to leverage the Swiss Grid activities

Represent the interests of the national Grid community towards other national and international bodies aims to become NGI to EGI

Activities organised in working groups: HEP provides strong input, e.g. ATLAS GRID is organised in SWING working group strong involvement of CSCS and SWITCH

see www.swiss-grid.org


Summary – Swiss ActivitiesSummary – Swiss Activities

Common operation of ONE single Swiss Tier-2

Reliably operates and delivers the Swiss pledges to the LHC experiments in terms of computing resources since Q2/2005

Growth in size as planned, final size reached ~ end 2009.

Compares well in size with other Tier-2s.

Tier-3 centres strongly complement Tier-2 :

operate in close collaboration – profit from know-how transfer.

overall size of all Tier-3 is about 100% CPU / 50 % disk of Tier-2

HEP is (still) majoriy community in GRID activity in CH.

We are prepared for PHYSICS !

HWW22


S.Gadomski, A.Clark (UNI Ge)

S.Haug, H.P. Beck (UNI Bern)

C.Grab (ETHZ) [ chair CCB ]

D.Feichtinger (PSI)

Z.Chen, U.Langenegger (ETHZ)

R.Bernet (UNIZH)

P.Szczypka, J. Van Hunen (EPFL)

Thanks to Tier-2 /-3 PersonnelThanks to Tier-2 /-3 Personnel

P.Kunszt (CSCS)

F. Georgatos, J.Temple, S.Maffioletti, R.Murri (CSCS)

and many more …


Optional slidesOptional slidesOptional slidesOptional slides


A.Clark, S.Gadomski (UNI Ge)

H.P.Beck, S.Haug (UNI Bern)

C.Grab (ETHZ) chair CCB

D.Feichtinger (PSI) vice-chair CCB

U. Langenegger (ETHZ)

R.Bernet (UNIZH)

J. Van Hunen (EPFL)

P. Kunszt (CSCS)

CHIPP Computing Board CHIPP Computing Board

Coordinates the Tier-2 activities representatives of all institutions and experiments


The Swiss ATLAS Grid The Swiss ATLAS Grid The Swiss ATLAS Grid The Swiss ATLAS Grid

• Swiss ATLAS GRID federation is based on: CSCS (T2) to T3s in Bern and Geneva For

• Total resources in 2009 ~800 cores and ~400 TB


ATLAS Tier3 at U. BernATLAS Tier3 at U. Bern ATLAS Tier3 at U. BernATLAS Tier3 at U. Bern

Hardware in production• In local cluster 11 servers

– ~30 worker CPU cores, ~30 TB disk storage

• In shared university cluster

• ~300 worker CPU cores (in 2009)

Upgrade plans (Q4 2009)• ~100 worker cores in local cluster

• Increased share on shared cluster

Usage• Grid site since 2005 (ATLAS production)

• Local resource for LHEP’s analyses and simulations

S. Haug ~ 30 / 6% (CPU/disk) size of Tier-2 ~ 30 / 6% (CPU/disk) size of Tier-2


ATLAS Tier3 at U. ATLAS Tier3 at U. GenevaGeneva

ATLAS Tier3 at U. ATLAS Tier3 at U. GenevaGeneva

Hardware in production • 61 computers

– 53 workers, 5 file servers, 3 service nodes

• 188 CPU cores in the workers

• 44 TB of disk storage

• Upgrade plans• grid Storage Element with 105 TB (Q1 2009)

• Advantages of the Geneva Tier3• environment like at CERN, latest ATLAS software

via AFS

• direct line to CERN (1 Gbps)

• popular with ATLAS physicists (~60 users)

S. Gadomski ~ 20/9% size of Tier-2 ~ 20/9% size of Tier-2


CMS : common Tier 3 at PSI CMS : common Tier 3 at PSI CMS : common Tier 3 at PSI CMS : common Tier 3 at PSI

Common CMS Tier-3 for ETH, PSI, UZH groups

in operation at PSI since Q4/2008

1 Gbps connection PSI ZH

Upgrade in 2009 by factor 2 planned.

Year 2008 2009

CPU / kSI2k 215 500

Disk / TB 105 250

~ 15/20% size of Tier-2~ 15/20% size of Tier-2 ~ 15/20% size of Tier-2~ 15/20% size of Tier-2

+2 more X4500

~ 25 users now, growingoperates GRID storage element anduser interface to enable users direct GRID access.

Local production jobs

D.Feichtinger


LHCb: Tier-3 at EPFLLHCb: Tier-3 at EPFLLHCb: Tier-3 at EPFLLHCb: Tier-3 at EPFL

Hardware and Software:Machines identical to those in the LHCb pit

58 Worker nodes x8 cores (~840 kSI2k) 36 Tb of storage

Uses SLC4 binaries of LHCb software andDIRAC Development builds

Current Status and Operation:

EPFL is one of the pilot DIRAC Sites

Custom DIRAC interface for batch access

Active development to streamline GRID usage

Aim to run official LHCb MC production

P. Szczypka ~ 50 / 7 % size of Tier-2~ 50 / 7 % size of Tier-2 ~ 50 / 7 % size of Tier-2~ 50 / 7 % size of Tier-2


Hardware in production

Zurich local HEP Cluster: small Intel Cluster for local LHCb jobs Hardware: CPU: 125 kSI2k, Disk: ~15 TB

Shared Zurich Matterhorn Cluster at IT-dept. of UZH: Only used for Monte Carlo Production (~ 25 kSI2k)

Replacement in Q3/2009 in progress

Usage:

local analysis resource

Monte Carlo Production for LHCb

LHCb : Tier-3 at U. ZurichLHCb : Tier-3 at U. ZurichLHCb : Tier-3 at U. ZurichLHCb : Tier-3 at U. Zurich

~ 10 / 5 % size of Tier-2~ 10 / 5 % size of Tier-2 ~ 10 / 5 % size of Tier-2~ 10 / 5 % size of Tier-2

by R. Bernet


Summary: Swiss Tier-3 Efforts Summary: Swiss Tier-3 Efforts (Q1/09)(Q1/09)

Nr cores CPU(kSI2k)

Storage (TB)

Comments

ATLAS BE GE

30+300s188

~600 320

3044

BE: GRID usage since 2005 for Atlas productionGE: identical SW-environment to CERN; direct line to CERN.

CMS ETHZ, PSI, UZH

72 ~250 105Operates GRID storage element and user interface to enable direct GRID access.

LHCb EPFL UZH

464shared

~800125

3615

EPFL is a DIRAC pilot site identical machines as in pit

UZH:MC production; shared

Total Tier-3 ~2100 230 cf: Tier-2: 1600 kSI2k, 520 TB

Tier-3 capacities : similar size in CPU as Tier-2 ; and ~ 50% diskSubstantial investment of resources for MC production+local analysis.(… more details in backup slides)

Note: CPU numbers are estimates; upgrades in progress …

Status 1.3.09


non-LHC GRID Effortsnon-LHC GRID Effortsnon-LHC GRID Effortsnon-LHC GRID Efforts

KnowARC: Grid-enabled Know-how Sharing Technology Based on ARC Services

and Open Standards" (KnowARC) ; Next Generation Grid middleware based on NorduGrid's ARC

CoreGRID : European Network of Excellence (NoE) aims at strengthening and advancing scientific and technological excellence in the area of Grid and Peer-to-Peer technologies.

GridCHEM: "Computational Chemistry Grid" (CCG) is a virtual organization that provides access to high performance computing resources for computational chemistry (mainly US)

DILIGENT: Digital Library Infrastructure on Grid Enabled Technology (6th FP)

SEPAC: Southern European Partnership for Advanced Computing grid project

PRAGMA: Pacific Rim Application and Grid Middleware Assembly (PRAGMA)

Swiss GRID Activities Christoph Grab Lausanne, March 2009 Lausanne, March 2009.

Documents

Transcript of Swiss GRID Activities Christoph Grab Lausanne, March 2009 Lausanne, March 2009.