3/27/2007Grid Efforts in Belle1 Hideyuki Nakazawa (National Central University, Taiwan), Belle...

3/27/2007 Grid Efforts in Belle 1

Grid Efforts in Belle

Hideyuki Nakazawa(National Central University, Taiwan),

Belle Collaboration, KEK


Out Line

Belle experiment Computing system in Belle LCG at KEK and Belle VO status Introduction of SRB Summary


KEKB Accelerator•Asymmetric e+e- collider•3.5 GeV on 8 GeV

•3 km circumference•22mrad Crossing Angle•Continuous InjectionBelle Detector•Generic purpose•7 sub-detectors

“B factory” experiment at KEK (Japan).

BelleKEKB

Linac

3km

Mt. Tsukuba

Belle Experiment

BBSee )4(


Belle Collaboration

13 countries, 57 institutes, ~400 collaborators

IHEP, ViennaITEPKanagawa U.KEKKorea U.Krakow Inst. of Nucl. Phys.Kyoto U. Kyungpook Nat’l U. EPF Lausanne Jozef Stefan Inst. / U. of Ljubljana / U. of MariborU. of Melbourne

Aomori U.BINPChiba U.Chonnam Nat’l U.U. of CincinnatiEwha Womans U.Frankfurt U.Gyeongsang Nat’l U.U. of HawaiiHiroshima Tech.IHEP, BeijingIHEP, Moscow

Nagoya U.Nara Women’s U.National Central U.Nat’l Kaoshiung Normal U.National Taiwan U.National United U.Nihon Dental CollegeNiigata U.Osaka U.Osaka City U.Panjab U.Peking U.U. of PittsburghPrinceton U.RikenSaga U.USTC

Seoul National U.Shinshu U.Sungkyunkwan U.U. of SydneyTata InstituteToho U.Tohoku U.Tohuku Gakuin U.U. of TokyoTokyo Inst. of Tech.Tokyo Metropolitan U.Tokyo U. of Agri. and Tech.Toyama Nat’l CollegeU. of TsukubaUtkal U.VPIYonsei U.

Lots of contribution from TaiwanLots of contribution from Taiwan


LuminosityProduce large amount of B mesons!!

peak luminosity1.7118 × 1034 cm-2s-1

710 fb-1

1 fb-1~106 BB

Inte

grat

ed L

umin

osit

y (f

b-1)

●Crab CavityCrab Cavity installed,installed, being tuned now.being tuned now. Luminosity doubled?Luminosity doubled?

Integrated Luminosity

1 fb-1 ~ 1TB / day


History of Belle computing system

Performance 1997-4 years

2001-5 years

2006-6 years

Computing Server[SPECint2000 rate]

~100(WS)

~1,250(WS+PC)

~42,500(PC)

Disk Capacity [TB]

~4 ~9 1000

Tape Library Capacity[TB]

160 620 3,500

Work Group Server[# of hosts]

3+(9) 11 80+16FS

User Workstation[# of hosts]

25WS+68X

23WS+100PC

128PC


Overview of the B Computer

Storage

ComputingServers

WorkgroupServers

reservedfor Grid

On-lineReconstructionFarm


Belle SystemBelle System

Computing Server: ~42,500 SPECint2KComputing Server: ~42,500 SPECint2KStorage System (DISK): 1PBStorage System (DISK): 1PB

Storage System (HSM): 3.5PBStorage System (HSM): 3.5PB


Data Production at Belle

onlinereconstructionfarm

““MDST” dataMDST” data (four vector, PID info etc.)(four vector, PID info etc.)

rawdata +rawdata +““DST” dataDST” data

production

Users' analyes

hadron 120TB+ others

~ 1PB

MCMC

Generation and

DetectorSimulation

2.5THz(to finish in6 months)

2THz(to finish in2 months)

HSM

non-HSM

Loose Loose SelectionSelection Criteria Criteria

@500/fb@500/fb


Why Grid in Belle? No urgent requirement No urgent requirement Belle shifts to precise and exotic measurementBelle shifts to precise and exotic measurement

More MC statistics necessary for precise More MC statistics necessary for precise measurementmeasurement

New skim for exotic processNew skim for exotic process Lesson in de facto standardLesson in de facto standard

Maybe we should Maybe we should start considering start considering

about Gridabout Grid

Just my feeling

Just my feeling


Grid Introduction Strategy Strong support from KEK CRCStrong support from KEK CRC Starting with MC production and Starting with MC production and

accumulating experiences, gradually accumulating experiences, gradually shift to handle experimental data shift to handle experimental data

RecruitmentRecruitment Some collaborators who have running Some collaborators who have running

LCG are preparing to join the Belle VOLCG are preparing to join the Belle VO Experiencing Grid potential may Experiencing Grid potential may

changechangeBelle’s recognitionBelle’s recognition??


LCG Deployment at KEKLCG Deployment at KEK

JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-01 Since Nov. 2005.Since Nov. 2005. Registered to GOC, in operatiRegistered to GOC, in operati

on as WLCGon as WLCG Site Role:Site Role:

practice for production systepractice for production system JP-KEK-CRC-02.m JP-KEK-CRC-02.

test use among university groups in Japtest use among university groups in Japan.an.

Resource and Component:Resource and Component: SL-3.0.5 w/ gLite-3.0 laterSL-3.0.5 w/ gLite-3.0 later CPU: 14, Storage: ~1.5TBCPU: 14, Storage: ~1.5TB FTS, FTA, RB, MON, BDII, LFC, CE, SEFTS, FTA, RB, MON, BDII, LFC, CE, SE

Supporting VOs:Supporting VOs: bellebelle, apdg, g4med, ppj, dteam, ops an, apdg, g4med, ppj, dteam, ops an

d aild ail

JP-KEK-CRC-02JP-KEK-CRC-02JP-KEK-CRC-02JP-KEK-CRC-02 Since early 2006.Since early 2006. Registered to GOC, in operation Registered to GOC, in operation

as WLCGas WLCG Site Role:Site Role:

More stable services based on KEKMore stable services based on KEK-1 experiences. -1 experiences.

Resource and Component:Resource and Component: SL or SLC w/ gLite-3.0 laterSL or SLC w/ gLite-3.0 later CPU: 48, Storage: ~1TB (w/o HPSCPU: 48, Storage: ~1TB (w/o HPS

S)S) Full componentsFull components

Supporting VOs:Supporting VOs: bellebelle, apdg, g4med, atlasj, ppj, ilc, , apdg, g4med, atlasj, ppj, ilc,

dteam, ops and aildteam, ops and ail

Operation is supported by great efforts by APOperation is supported by great efforts by APROC members in ASGC.ROC members in ASGC.Operation is supported by great efforts by APOperation is supported by great efforts by APROC members in ASGC.ROC members in ASGC.


Belle VOBelle VO 9 sites　 Belle software are installed to 3 sites (KEK x2, ASGC)

~60 CPUs 2TB storage MC production ongoing

Installation manual ready GFAL with Belle software


Total Number of Jobs at KEK in 2006Total Number of Jobs at KEK in 2006

JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-01 JP-KEK-CRC-02JP-KEK-CRC-02JP-KEK-CRC-02JP-KEK-CRC-02

200200

700700

400400

1,0001,000

1,4001,400

BelleBelleBelleBelleBelleBelleBelleBelle


Total CPU Time at KEK in 2006Total CPU Time at KEK in 2006(Normalized by 1kSI2K)(Normalized by 1kSI2K)

JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-01 JP-KEK-CRC-02JP-KEK-CRC-02JP-KEK-CRC-02JP-KEK-CRC-02

4,0004,000

3,0003,000

1,000 [hrs kSI2K]1,000 [hrs kSI2K]

12,00012,000

10,00010,000

4,0004,000

BelleBelleBelleBelleBelleBelleBelleBelle


Logical Site OverviewLogical Site Overview

KEK FirewallKEK Firewall SuperSINETSuperSINETSuperSINETSuperSINET

HSMHSMHSMHSM

Grid LANGrid LAN

KEK-2KEK-2202.13.197.0/24202.13.197.0/24

KEK-2KEK-2202.13.197.0/24202.13.197.0/24

KEK-DMZKEK-DMZ

MCATMCAT172.22.28.0/24172.22.28.0/24

MCATMCAT172.22.28.0/24172.22.28.0/24

130.87.224.0/21130.87.224.0/21

SRBSRB172.22.28.0/24172.22.28.0/24

130.87.224.0/21130.87.224.0/21

SRBSRB172.22.28.0/24172.22.28.0/24

SRB-DSISRB-DSI130.87.104.0/22130.87.104.0/22

SRB-DSISRB-DSI130.87.104.0/22130.87.104.0/22

KEK-CCKEK-CC

KEK-1KEK-1130.87.208.0/22130.87.208.0/22

KEK-1KEK-1130.87.208.0/22130.87.208.0/22

$ scp output Belle:$ scp output Belle:

$ scp input Grid:$ scp input Grid:

Local files CPUsCPUs

WSWS


SRB Introduction ScheduleSRB Introduction Schedule

Construction PlanningConstruction Planning GridGrid Belle OperationBelle Operation NetworkingNetworking KEKCC/IBMKEKCC/IBM

Construction PlanningConstruction Planning GridGrid Belle OperationBelle Operation NetworkingNetworking KEKCC/IBMKEKCC/IBM

MCATMCATMCATMCAT

SRBSRBSRBSRB

FWFWFWFW

SRB-DSISRB-DSISRB-DSISRB-DSI

TestTestTestTest

ConnectionConnectionConnectionConnection

Start OperationStart OperationStart OperationStart Operation

PreparationPreparation


Belle Grid Deployment Future PlanBelle Grid Deployment Future Plan Federate with Japanese universities.Federate with Japanese universities.

KEK hosts the Belle experiment and behaves as Tier-0.KEK hosts the Belle experiment and behaves as Tier-0. Univ. with reasonable resources: full LCG (Tier-1)Univ. with reasonable resources: full LCG (Tier-1) Univ. without resources: UIUniv. without resources: UI

The central services such The central services such as VOMS, LFC and FTS as VOMS, LFC and FTS are provided by KEK. are provided by KEK.

KEK also covers web KEK also covers web Information and support Information and support service.service.

Grid operation is co-Grid operation is co-operated with 1~2 staffs operated with 1~2 staffs in each full LCG site.in each full LCG site.

JP-KEK-CRC-02JP-KEK-CRC-02 JP-KEK-CRC-03JP-KEK-CRC-03

UniversityUniversityUIUI



UniversityUniversity

UIUIUniversityUniversity





UIUI

Tier-0Tier-0

Tier-1Tier-1

deploy in the futuredeploy in the future

preliminary designpreliminary design


Summary

Belle VO launchedBelle software are installed to 3

sites KEK sites are mainly used by Belle

MC production ongoingSRB is being introduced


Additonal (Belle's) Resources

We now obtain high-performance computer system;but we didn't suddenly switch to the “less expensive” system.

350TB disks1.5PB tapes

934 CPUs

20units/20TB

We have been testing suchsystem for several years.

●Linux based PC clusters●S-ATA disk based RAIDdrives

●S-AIT tape drives

1000TB disks3.5PB tapes

2280 CPUs B computerfor comparison

These resources have been essentialfor Belle (production/analysis)


Belle Grid Deployment PlanBelle Grid Deployment Plan

We are planning a 2-phased deployment for BELLE experimWe are planning a 2-phased deployment for BELLE experiments.ents. Phase-1: BELLE user uses VO in JP-KEK-CRC-02 sharing with oPhase-1: BELLE user uses VO in JP-KEK-CRC-02 sharing with o

ther VOs.ther VOs. JP-KEK-CRC-02 consists of “JP-KEK-CRC-02 consists of “Central Computing SystemCentral Computing System” maintaine” maintaine

d by IBM corporation.d by IBM corporation. Available resources:Available resources:

CPU: 72 processors (opteron), SE: 200TB (with HPSS)CPU: 72 processors (opteron), SE: 200TB (with HPSS) Phase-2: Deployment of JP-KEK-CRC-03 as BELLE Production Phase-2: Deployment of JP-KEK-CRC-03 as BELLE Production

SystemSystem JP-KEK-CRC-03 uses a part of “JP-KEK-CRC-03 uses a part of “B Factory Computer SystemB Factory Computer System” resour” resour

ces.ces. Available resources (maximum estimation)Available resources (maximum estimation)

CPU: 2200 CPU,CPU: 2200 CPU, 　　 SE: 1PB (disk), 3.5 PB (HSM)SE: 1PB (disk), 3.5 PB (HSM) This system will be maintained by CRC and NetOne corporation.This system will be maintained by CRC and NetOne corporation.


Computing Servers

●DELL Power Edge 1855Xeon 3.6GHz x2Memory 1GB

●Made in Taiwan [Quanta]●WG: 80 servers (for login)Linux (RHEL)

●CS: 1128 serversLinux (CentOS)

●total: 45662 SPEC CINT2000 Rate.equivalent to 8.7THz

CPU will be increased by x2.5 (i.e. to 110000 SPEC CINT2000 Rate) in 2009.

1 enclosure = 10 nodes / 7U space1 rack = 50 nodes


Storage System (Disk)●Total 1PBwith 42 file servers(1.5PB in 2009)

●SATAII 500GB diskx ~2000(~1.8 failure/day ?)

●3 types of RAID(to avoid problems)

●HSM = 370 TBnon-HSM = 630 TB

ADTX ArrayMasStor LP15drive/3U/7.5TB

Nexan SATA Beast42drive/4U/21TB

SystemWorksMASTER RAID B123016drive/3U/8TB(made in Taiwan)


Storage System (Tape)

●Backup●90TB + 12drv + 3srv●LTO3 400GB/volume●NetVault

●HSM: PetaSite (SONY)●3.5PB + 60drv + 13srv●SAIT 500GB/volume ●30MB/s drive●Petaserve

3/27/2007Grid Efforts in Belle1 Hideyuki Nakazawa (National Central University, Taiwan), Belle...

Documents

Transcript of 3/27/2007Grid Efforts in Belle1 Hideyuki Nakazawa (National Central University, Taiwan), Belle...