1 DataGRID Application Status and plans [email protected].

1

DataGRID Application

Status and plans

[email protected]

2DataGRID ConferenceMarch 6, 2002

Application objectives for year 1

Define use cases for the applicationsDefine application requirementsDeploy realistic applications on the TestBed 1Evaluate TestBed 1


AchievementsDialogue within the applications on the opportunities of GRID

Dialogue and coordination to understand how to do things differently and not more of the same

Testbed 0 has improved our understanding of the issues A very large amount of unfunded application effort has made

this possible Realistic applications using GLOBUS have been deployed

Requirements expressed in June 2001 Developed detailed “long term use cases” A very large exercise leveraging unfunded effort (30-40

person-month)Funded WP8 effort (LCs) has

Participated to the deployment and configuration of the TestBed

Validated the TestBed with “generic” HEP applications Resource Broker Replica catalogue functionality


AchievementsDetailed TestBed evaluation plans have been elaborated and applied

Few hours after the official opening of the TestBed on December 9 a standard physics simulation job has been run

Although TB1 had not reached stability much middleware functionality was demonstrated by the applications in a short period of time


Achievements

Mechanisms to provide feedback to the developers have been put in place Priority list constantly updated by WP8 and

discussed at weekly WP manager meetings Feedback on the release plan Detailed user requirements for crucial

components (e.g. Storage Element) in preparation

Large unfunded participation from experiments Probably as much as 200 person-month in the

first year


Issues and actionsStability (or even existence) of the testbed

Testbed existed for 10 days in December and then for ~5 days before the review!!

Configuration and management of the VO’s is very difficult

Our needs were not expressed too clearly and were not given enough attention

Procedures should be worked out to automate it

Assignment and escalation of problems is complicated

It has worked well thanks to the heroic efforts of all people involved, but it will have to be better formalised

Need tight coordination between site-managers, TestBed and applications


Issues and actionsCoexistence of pseudo-production and development environment still a problem

We have done it for years on a local environments On a distributed ones is more difficult and we are

learning

Deployment plans to all TestBed sites are still unclear

This needs to be clarified!


Plan for the next yearContinue exploitation of TB1

Deploy larger and more realistic applications on TB1, including “data challenges” and “production challenges”

Use more sites as TB1 expands

Refine the user requirements and long term views A common set of requirements / use cases to define

common solutions This will facilitate the relation with other GRID MW projects Collaboration with DataTag has MW interoperability as

specific goal An RTAG has been launched in the LCG Project with this

mandate This has been possible because of the work of the DataGRID

project and is heavily based on WP8's findings to date


What we want from a GRIDThis is the result of our experience on TB0 & TB1

OS & Net services

Basic Services

High level GRID middleware

LHCVO common application layer

Other apps

ALICE ATLAS CMS LHCb

Specific application layer

Other apps

GLOBUS team

GRID architectur

e


Common use cases

OS & Net services

Bag of Services (GLOBUS)

Specific application layer

GLOBUS team

MiddleWare

MW1 MW2 MW3 MW4 MW5

MiddleWare

MW1 MW2 MW3 MW4 MW5

LHCVO use cases & requirements Other apps

If we manage to define

ALICE ATLAS CMS LHCb Other apps

MiddleWare

MW1 MW2 MW3 MW4 MW5

VO use cases & requirements Common core use case

Or even better

LHC Other apps

MiddleWare

MW1 MW2 MW3 MW4 MW5

VO use cases & requirements

It will be easier to arrive at

Common use cases

LHC Other apps


Detailed plan of workLCs have already interviewed the experiments

A lot of material has been collectedThey will now post-process the information

Hopefully one monthThe result will be the starting point for the RTAG

RTAG will contain representative of experiments and (subject to the agreement of the experiments) of LCs

The RTAG produces results A of detailed use cases for HEP in two-three months

During this time LCs will devote their attention to the testbed (see later)The results will be brought back into WP8/9/10 to

Discuss possible implementation Involve WP9-10


Production Scenario (PRO)

VO MetaDataCatalogLFN1LFN2LFN3LFN4...

JDLWDL file Broker

Production db

UIExp.Frame

VO-ReplicaCatalog

CE

SE

WNConfiguration db GIIS

GRIS

MSS


VO MetaDataCatalogLFN1LFN2LFN3LFN4...

JDL file

Broker

Production db VO-ReplicaCatalog

CE

SE

WNConfiguration db

Rcapifile open

UIExp.Frame

SE

Spawn a job

EvtTagCollection

Physics Analysis (ANA) scenario

Retrieve result


Execution (additional data/info needed)

Job on worker node

Catalog(LFN)

Calibration…

ReplicaCatalogue

InformationServices

Scheduling Grid


Postprocessing

Job on worker node

Catalog(LFN)

Calibration…

ReplicaCatalogue

InformationServices

Storage Element

SEn


Disclaimer

A lot of this work is essentially architectural We want to provide a purely user point of view Inevitably we will trespass the border and say

not only what we want but how we want it

However we understand this and we will be glad to be suggested different ways to do that As long as this still does what we want


Application PrioritiesAPI’s

This is a long standing requirement, now this is needed for prototyping

MPI At least on local clusters, wide area MPI later

Interactive access We have to agree on what it means and how it can be

provided We certainly want to start trying on TB1

Linux 7.2 (summer) This is necessary for a proper deployment

Solaris (release 2) Not a priority per se, but if not this year, we will never do it The DataGRID code should demonstrate some portability to

be seriously considered as a baselineTools for VO management


Application testbed requirements

Three main priorities


Application testbed requirements

Development and production testbeds We need a valid model to have both

Consolidation & Stability of production testbed It has to stay up from tomorrow to release 2Consolidation and complete deployment of uniform information system

The question of FTREE/R-GMA/MDS has to be solved

Support for the testbed This has to work much better than in TB1 till now

Interface to MSS CASTOR, HPSS, DataSTORESolve the question of installation on pre-existing clusters

Light LCFG?Expansion of the testbed

(up to) the famous 40 WP6 sites


Immediate consolidationMay be jump started by a jam session of LCs and WPs Get together for few days till some test

applications run on the testbed without problems

This should be done rather quickly 3-4 days week 22th March or after Easter

These applications may be bundled in a standard test package to run periodically A kind of testbed heartbeat to verify its

stability


SummaryApplications have been able to deploy and demonstrate large applications on the TestBed

The LHC experiments are participating with enthusiasm to the project

The side effect is that the pressure is high from the users for

Support Stable environment Functionality Documentation Wide deployment

These are healthy signs Continued positive reaction to the feedback from the users

will maintain their high level of enthusiasm and participationThe evolution of TB1 leading to TB2 will be a crucial test of the project’s ability to reach its final goals

1 DataGRID Application Status and plans [email protected].

Documents

Transcript of 1 DataGRID Application Status and plans [email protected].