HEP Data GRID in Japan

HEP Data GRID in JapanHEP Data GRID in Japan

Yoshiyuki WataseYoshiyuki Watase

Computing Research Center Computing Research Center

KEKKEK

HEP Data Grid Workshop Nov. 8-9 2002at

CHEP KNU Korea

KEK High Energy Accelerator Research Organization

22Nov. 8 2002Nov. 8 2002 HEP Data Grid Workshop @ KNUHEP Data Grid Workshop @ KNU

OutlineOutline1.1. Network Infrastructure Network Infrastructure 2.2. HEP Grid Activities in JapanHEP Grid Activities in Japan

1.1. ATLASATLAS2.2. BelleBelle3.3. R&D of GFarmR&D of GFarm4.4. Related “Virtual Laboratory” activitiesRelated “Virtual Laboratory” activities

1.1. CollaboratoryCollaboratory2.2. GAN: Global Accelerator NetworkGAN: Global Accelerator Network

3.3. Funding status for GridFunding status for Grid4.4. ConclusionConclusion


Network Infrastructure 1Network Infrastructure 1

SuperSINETSuperSINET (Science Network (Science Network)) Jan. 2002, fiber optic network was provided by Jan. 2002, fiber optic network was provided by NIINII for r for r

esearch in Univ.’s and Institutes ( ~ up to 40 nodes).esearch in Univ.’s and Institutes ( ~ up to 40 nodes). 10 Gbps IP backbone and many p-to-p GbE links for re10 Gbps IP backbone and many p-to-p GbE links for re

search groups: search groups: HEP, Astronomy, Bioinformatics, Nano-tech, GRID coHEP, Astronomy, Bioinformatics, Nano-tech, GRID computingmputing

HEP HEP MPLS-VPN of HEPnet-J for ~25 university groupsMPLS-VPN of HEPnet-J for ~25 university groupsby sharing the 10 Gbps IP backboneby sharing the 10 Gbps IP backboneGbE links from KEK to Univ. ‘sGbE links from KEK to Univ. ‘sTohoku, Tokyo, Nagoya, Kyoto, Osaka, Tsukuba, Tohoku, Tokyo, Nagoya, Kyoto, Osaka, Tsukuba,

Titech(Tokyo Inst. of Tech), ICRR(Inst. Cosmic Ray Research)Titech(Tokyo Inst. of Tech), ICRR(Inst. Cosmic Ray Research)

NIINII: National Institute for Informatics: National Institute for Informatics


Nagoya Hub

Internet

Osaka U

Kyoto U

Nagoya U

NIFS

NIG

KEK

Tohoku U

IMSU-Tokyo

U Tokyo

NII

IP 10 GbpsWDM path

IP router

OXC

Tokyo HubOsaka Hub

ICRKyoto-U

NII Chiba

US， EU

Waseda

TITEC

Tsukuba

Hokkaido

IMS

Kyushu　 UKorea

NAO

ISAS

Network Topology of SuperSINET


Network Infrastructure 2Network Infrastructure 2

East bound: to US, EUEast bound: to US, EU NII will implements 2.4 G x 2 terminated at NY on Jan. 2003.NII will implements 2.4 G x 2 terminated at NY on Jan. 2003. We need transit line between NY and StarLight for connection to We need transit line between NY and StarLight for connection to

CERN with end-to-end GbE.CERN with end-to-end GbE. We need peering to ESnet at NY.We need peering to ESnet at NY.

West bound: to KoreaWest bound: to Korea Genkai project will provide ~ GbE connection between CHEP(KGenkai project will provide ~ GbE connection between CHEP(K

NU) and KEK.NU) and KEK. Being demonstrated at APAN Fukuoka Meeting Jan. 22, 2003Being demonstrated at APAN Fukuoka Meeting Jan. 22, 2003

Nov. 8 2002 HEP Data Grid Workshop @ KNU 6

Network Infrastructure

NII- NY

TRANSPAC

Taiwan-USJapan (NII) -NY : 2.4G x 2 Jan. 2003Japan – US: 622M x 2 (TRANSPAC)Korea – US: 45 MKorea – Japan: 2.4G Jan. 2003Korea – Europe: 2 M (TEIN)China(IHEP) – Japan(KEK): 128 kbps (HEP)China – US: 10 MTaiwan – Japan: 155 MTaiwan – US: 622 M (Dec. 2002)

TEIN


LHC/Atlas-1LHC/Atlas-1Atlas Regional CenterAtlas Regional Center Being set up at ICEPP Being set up at ICEPP (Int’l Center for Particle Physics, U. (Int’l Center for Particle Physics, U.

Tokyo)Tokyo) Installation of PC farms and StorageInstallation of PC farms and Storage

PC farm: 78 x P-III 1.4GHzPC farm: 78 x P-III 1.4GHz

+ New PC farm :214 x Xeon 2.8 GHz (early 2003)+ New PC farm :214 x Xeon 2.8 GHz (early 2003)

Tape TLO 7TB ( CASTOR)Tape TLO 7TB ( CASTOR)

+ 20-30 TB Disk data server (early 2003)+ 20-30 TB Disk data server (early 2003)

Data Challenge DC1Data Challenge DC1Phase-1 (~ Sep. ): Monte Carlo production 5x10Phase-1 (~ Sep. ): Monte Carlo production 5x107 7 eventsevents

Phase-2 ( Nov. ~ ) : Generate events with event-overlappingPhase-2 ( Nov. ~ ) : Generate events with event-overlapping


LHC/Atlas -2LHC/Atlas -2Globus installed in PC’s at KEK and ICEPPGlobus installed in PC’s at KEK and ICEPP

Testing Globus environmentsTesting Globus environments

Grid environments for LHC exp. :Grid environments for LHC exp. : LCG – 1 LCG – 1 Start test on Nov. 2002Start test on Nov. 2002 Testing between ICEPP and KEKTesting between ICEPP and KEK Testing between ICEPP and other sites in EU/US ( 2003 )Testing between ICEPP and other sites in EU/US ( 2003 )

Remote data access via GbERemote data access via GbE Atlas simulation data at ICEPP being stored in HPSS storAtlas simulation data at ICEPP being stored in HPSS stor

age at KEK through GbE linkage at KEK through GbE link Test Resources: ICEPP PC farm 78 cpu’sTest Resources: ICEPP PC farm 78 cpu’s

KEK PC farm 100cpu’s KEK PC farm 100cpu’s HPSSS ~ 20 TB HPSSS ~ 20 TB


LHC/Atlas - 3LHC/Atlas - 3KEK

ICEPP, Tokyo

GbE (~60 km)

simulation data ~ 10 TB

0

200

400

600

800

1000

0 250 500 750 1000

Window Size (kbytes)

(M

bps)

スル

ープ

ット

ICEPP→ KEK(Stream1)ICEPP→ KEK(Stream2)ICEPP→ KEK(Sum)KEK→ ICEPP(Stream1)KEK→ ICEPP(Stream2)KEK→ ICEPP(Sum)

Data Transfer Throughput Test over GbE 1. netperf test 2. ftp 3. HPSS access over GbE

Local access ~ 30 MB/sec WAN access ~ being measured

IBM RS6000/SP HPSSPC Farms PC Farm

netperf test over 2 streams

Single stream netperf: ~70 MB/s ftp : ~ 30MB/s

Data Transfer Throughput Test


KEKB/BelleKEKB/BelleB Factory experimentB Factory experiment

Luminosity reached to 8 x 10Luminosity reached to 8 x 103333 /cm /cm22/s in Oct. 2002: updating the world rec/s in Oct. 2002: updating the world record and still improving ord and still improving

Data storage rate = ~ 500 GB/dayData storage rate = ~ 500 GB/day

Data analysisData analysis Resources at KEK: Resources at KEK:

1000 PC’s for production and simulation1000 PC’s for production and simulationStorage ~ 630 TBStorage ~ 630 TB

Resources at Univ.’sResources at Univ.’sNagoya Tokyo TohokuNagoya Tokyo TohokuSmaller resources in the collaboration institutes compared to BaBar eSmaller resources in the collaboration institutes compared to BaBar exp.xp.

BaBar has partners in IN2P3, RAL.BaBar has partners in IN2P3, RAL.Need more funding for universitiesNeed more funding for universities

planning to extend their computing power in Data Grid parplanning to extend their computing power in Data Grid paradigmadigm


KEKB/Belle:KEKB/Belle: Data Moving for Data Analysis over SuperSINET GbEData Moving for Data Analysis over SuperSINET GbE

U. Tokyo

KEK

Osaka U.

Nagoya U.

Neutrino CenterTohoku U.

1TB/day~100Mbps

~ 100GB/day400 GB/day~45 Mbps

170 GB/day

NFS

e+e- Bo Bo


R&D of Gfarm(Grid data farm)R&D of Gfarm(Grid data farm)GfarmGfarm http://datafarm.apgrid.orhttp://datafarm.apgrid.org/g/ Development by Development by AISTAIST, Titech, KEK , Titech, KEK since 2000since 2000 ArchitectureArchitecture

PC farm with large local disk/node as a GRID fabricPC farm with large local disk/node as a GRID fabricLarge data file is divided into fragments and stored in the disks Large data file is divided into fragments and stored in the disks by read-inby read-inData file integrity is managed by the Gfarm metadata DB Data file integrity is managed by the Gfarm metadata DB Data I/O by parallel file systemData I/O by parallel file systemAffinity scheduling of process Affinity scheduling of process and storage for data residence and storage for data residence Service daemon process: Service daemon process: gfsd is running at each nodegfsd is running at each nodeAuthentication by gfarm key Authentication by gfarm key and/or GSI and/or GSI AIST AIST ::Advanced Inst. for Science and TechnologyAdvanced Inst. for Science and Technology


Gfarm TestGfarm Test PC farm used: Athlon 1.2GHz 16nodesPC farm used: Athlon 1.2GHz 16nodes

Fast EthernetFast Ethernet Data replication between two 8-nodesData replication between two 8-nodes

8 parallel copy of 1GB fragments : 8 GB data file8 parallel copy of 1GB fragments : 8 GB data file

% gfrep –H <nodes> gfarm:userdata.bin% gfrep –H <nodes> gfarm:userdata.bin

Throughput 89.4 MB/s vs. 90.2 MB/s (sum of independent transfThroughput 89.4 MB/s vs. 90.2 MB/s (sum of independent transfers)ers) Overhead in access to the metadata ~ 1% Overhead in access to the metadata ~ 1%

Metadata DB Server

Gfarm <nodes>Data fileData file gfarm:userdata.bingfarm:userdata.bin


Gfarm Demonstration at SC2002 Nov. 18“Bandwidth Challenge”

Data transfer, data replication over trans-oceanic network

NationalBackbone

NationalBackbone

Super SINET

Super SINET

Cluster and Network setting for SC2002 Bandwidth Challenge (9/3)

SCinetNOC

3com 4924

10 Gbps

By courtesy ofForce10

SC2002, Baltimore

Indiana Univ.Cluster

SDSCCluster

IndianapolisGigaPoP

NOC

Toky oNOC

OC-12 x 2

Japan

TransPAC

KEKCluster

TitechCluster

AISTCluster

ICEPPCluster

AISTM160

Tsukuba-U

MaffinM160

GbE

10 GbE

GbE

Star LightPacif ic NW Gigapop

GbE

NOC

AISTBooth

Alternative

Tsukuba WAN

Abilene

GbE -> ?10-12 nodes5 TB, 1 GB/ s

HP4000

15 nodes180 GB

4-6 nodes2 TB, 400 MB/ s

Abilene

NII


Related activity : CollaboratoryRelated activity : Collaboratory““Virtual Laboratory” for Material ScienceVirtual Laboratory” for Material ScienceFunded 2001-2005 Funded 2001-2005 5 laboratories: 5 laboratories: KEK, KEK, IMSIMS, U. Tokyo, Tohoku U., Kyoto U. , U. Tokyo, Tohoku U., Kyoto U.

ImplementationImplementation Remote control of experimental equipmentsRemote control of experimental equipments

X-ray diffractometers at Photon Factory of KEK X-ray diffractometers at Photon Factory of KEK Data sharing Data sharing Simulation by supercomputers linkedSimulation by supercomputers linked IP video conferencing system with application sharingIP video conferencing system with application sharing

Demonstrated the prototype system Oct. 23, 2002Demonstrated the prototype system Oct. 23, 2002 Remote control from IMS(Nagoya) of diffractometers at the beaRemote control from IMS(Nagoya) of diffractometers at the bea

m line of PF/KEK.m line of PF/KEK. H.323 video conferencing with 4 other institutesH.323 video conferencing with 4 other institutes

IMSIMS: Institute for Molecular Science: Institute for Molecular Science


Related activity: GANRelated activity: GANGlobal Accelerator NetworkGlobal Accelerator Network

Aimed to remote operation of large accelerator facility, suAimed to remote operation of large accelerator facility, such as future Linear Colliderch as future Linear Collider

It facilitates acc control and monitoring for remote operatiIt facilitates acc control and monitoring for remote operation shifton shift

International workshop held 2001, 2002International workshop held 2001, 2002

KEK Planning to implement at J-PARC( KEK Planning to implement at J-PARC( Japan Proton Japan Proton

Accelerator Research ComplexAccelerator Research Complex)) 50 GeV, 3 GeV proton accelerators at Tokai 50 km away from Ts50 GeV, 3 GeV proton accelerators at Tokai 50 km away from Ts

ukuba(KEK)ukuba(KEK) Construction 2001 – 2006Construction 2001 – 2006 JAERI

Tokai CampusKEK


Funding status for GridFunding status for Grid

Gigabit Network Gigabit Network MEXTMEXT: Ministry of Edu, Sci, ..: Ministry of Edu, Sci, ..

MEXTMEXT Informatics Project Informatics Project A05: AA05: Application Grid projectspplication Grid projects for science ~ US$ 6M / 5 y for science ~ US$ 6M / 5 y

Networking supercomputers: Networking supercomputers: ITBLITBL ~ US$ 105M ~ US$ 105M / 5y / 5yAIST GTRC(Grid Technology Research Center)AIST GTRC(Grid Technology Research Center)

~ US$ 140M ~ US$ 140M

Other Funding from Institutional ResourcesOther Funding from Institutional Resources LHC/Atlas(ICEPP,KEK), GLHC/Atlas(ICEPP,KEK), Gfarm, farm, Virtual Observatory, Virtual Observatory, Ninf-G, Ninf-G,

BioGridBioGrid((Osaka U Osaka U )), OBIGrid, OBIGrid(Riken)(Riken),,…….…….

Projects proposed (2003 ~ 5 years)Projects proposed (2003 ~ 5 years) National Research Grid InitiativeNational Research Grid Initiative (US(US$$ 360M) 360M): focused in : focused in

Bio & NanoBio & Nanotechtech

National ‘Business’ Grid ProjectNational ‘Business’ Grid Project (US(US$$ 250M) 250M): ASP business: ASP business


Gigabit networkGigabit network

Super SINETSuper SINET 1010 Gbps MEXT Gbps MEXT DDWDM nationwide networkWDM nationwide network About About 4040 universities and labs universities and labs USUS$$ 60M/yr 60M/yr 55 application areas: HEP, Nanotech, application areas: HEP, Nanotech,

Bioinformatics, AstronomyBioinformatics, Astronomy, Grid Computing, Grid Computing

Tsukuba-WANTsukuba-WAN 330Gbps DWDM ring in the Tsukuba area330Gbps DWDM ring in the Tsukuba area Tens of national labsTens of national labs 6 supercomputers in TOP1006 supercomputers in TOP100 Testbed for Grid infrastructure and applicationsTestbed for Grid infrastructure and applications


MEXT Grid ProjectsMEXT Grid Projects• MEXT “Informatics” Project MEXT “Informatics” Project

A05 “Grid” Area (US$ 6M, 5 years)A05 “Grid” Area (US$ 6M, 5 years)– Lead by Shinji Shimozo(Osaka U.) & Satoshi Matsuoka(Titech)Lead by Shinji Shimozo(Osaka U.) & Satoshi Matsuoka(Titech)– 11 groups of universities and labs11 groups of universities and labs– Research and development on application, middleware, and inResearch and development on application, middleware, and in

frastructurefrastructure– Gfarm project is partly supported.Gfarm project is partly supported.

• Tokyo Institute of Technology “Titech Campus Grid” (UTokyo Institute of Technology “Titech Campus Grid” (US$ 2M)S$ 2M)

– Lead by Satoshi Matsuoka(Titech)Lead by Satoshi Matsuoka(Titech)– Total of 800 CPU PC clusters at 13 locations in a 30km ranTotal of 800 CPU PC clusters at 13 locations in a 30km ran

ge connected by Super TITANET(1-4Gbps) backbonege connected by Super TITANET(1-4Gbps) backbone– Pilot project for virtual parallel computer infrastructure for high Pilot project for virtual parallel computer infrastructure for high

end computational e-Science end computational e-Science


ITBLITBLITBL(IITBL(ITT-based Laboratory)-based Laboratory)

Government LabsGovernment Labs: NAL, RIKEN, NIED, NIMS, JST, JAERI: NAL, RIKEN, NIED, NIMS, JST, JAERI Project period: 2001-2005 (3-stage project) with total of USProject period: 2001-2005 (3-stage project) with total of US$$

105M funding105M funding Applications: mechanical simulation, computational biology, Applications: mechanical simulation, computational biology,

material science, environment, earthquake engineeringmaterial science, environment, earthquake engineering StepStep 1: 1: SSupercomputer centers upercomputer centers of gof government lab overnment lab areare

networknetworkeded via SuperSINET via SuperSINET Step 2: “Virtual Research Environment”: Grid-enabling Step 2: “Virtual Research Environment”: Grid-enabling

laboratory applicationslaboratory applications StepStep 3: Sharing information among researchers from widely 3: Sharing information among researchers from widely

distributed disciplines and institutionsdistributed disciplines and institutions


Grid Computing Applicationsimplementing key technologies of ITBL

J apan Meteorological Agency Numerical Weather Prediction Data

Atmospheric Dispersion Simulations

case1 case2 ......... case4

3D Wind Field Data

Wind Field Calculation

Prediction Data at Monitoring Points

Statistical Analysis

Estimation Result

Several hundreds of the simulationsbased on the possible releaseparameters are conducted quicklyby using parallel computers.

ObservationObservation Data Data

Real Time Viz. Multi‐Vision

Radioactive SourceEstimation System

Two-DimeTwo-Dimensional Data at Sea Surfaceat Sea Surface

StampiStampi StampiStampi

Two-Dimensionalonal Data Data at Ground SurfaceSurface

Environmental Circulation Simulationfor Pollutant Materials

VPP300VPP300（（ Vector Parallel ComputerVector Parallel Computer ））

Atmospheric EnvironmentAtmospheric EnvironmentSimulationSimulation

COMPAQ αCOMPAQ α（（ High-Performance PC)High-Performance PC)

Marine Environment Marine Environment SimulationSimulation

AP3000AP3000（（ Scalar Parallel ComputerScalar Parallel Computer ））

Terrestrial EnvironmentTerrestrial EnvironmentSimulationSimulation

Ion ParticlesIon Particles

StampiStampi

Vector MachineVector Machine Scalar MachineScalar Machine

Fluid-Particle Hybrid Simulationfor Tokamak Plasmas

Electronic fluidElectronic fluid/Electro-Magnetic field/Electro-Magnetic field

SPring8

ControlControl

DiagonalizationDiagonalization

OrthonormalizarionOrthonormalizarion

Pool of task distributionPool of task distribution

Vector MachineVector Machine Scalar MachineScalar Machine

Integral handlingIntegral handling

Partial accumulationPartial accumulation

FFijij<-F<-Fijij+D+Dklkl*q*qijklijkl

Large-scale Hartree-Fock Large-scale Hartree-Fock CalculationCalculation


Grid Technology Research CenterGrid Technology Research Center

Part of AIST under METI (Minitry of Economy, Trade and Part of AIST under METI (Minitry of Economy, Trade and Industry) Industry)

GTRCGTRC (Grid Technology Research Center)(Grid Technology Research Center) Established in Jan. 2002Established in Jan. 2002 Director: Satoshi SekiguchiDirector: Satoshi Sekiguchi

US$140M US$140M for building, equipment, R&D and operationsfor building, equipment, R&D and operations

To be central institute for Grid research in JapanTo be central institute for Grid research in Japan Not just R&D, but serve Not just R&D, but serve asas central CA, ApGrid central CA, ApGrid’s’s principal site principal site Collaboration with institutions and other Grid projectsCollaboration with institutions and other Grid projects of both of both

domestic and international scaledomestic and international scale Organizing GRID Consortium JapanOrganizing GRID Consortium Japan


ConclusionsConclusions

Grid activities are starting. But not coordinated suGrid activities are starting. But not coordinated support for middleware developmentpport for middleware developmentInternational testbed for LHC/Atlas in 2003International testbed for LHC/Atlas in 2003Possible collaboration in Asian region for KEKB/BePossible collaboration in Asian region for KEKB/BellelleNetwork is emerging for heavy application users.Network is emerging for heavy application users. domestic and international usedomestic and international use

Testbed of the Grid middleware in collaboration witTestbed of the Grid middleware in collaboration with CS peopleh CS people

HEP Data GRID in Japan

Documents

Transcript of HEP Data GRID in Japan