David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting...
-
Upload
ashlynn-adams -
Category
Documents
-
view
212 -
download
0
Transcript of David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting...
![Page 1: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/1.jpg)
David Adams
ATLAS
ADA, ARDA and PPDG
David AdamsBNL
June 28, 2004
PPDG Collaboration MeetingWilliams Bay, Wisconsin
![Page 2: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/2.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 2
David Adams
ATLAS
Contents
Definitions
ADA/DIAL model
Current status of ADA
Connections to PPDG
Conclusions
![Page 3: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/3.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 3
David Adams
ATLAS
DefinitionsADA – ATLAS Distributed Analysis
• ATLAS project to deliver an end-to-end analysis system
ARDA – A Realisation of Distributed Analysis for LHC• CERN project to coordinate the development of prototype
distributed analysis systems for the LHC experiments
• Prototypes are based on the EGEE middleware (gLite)
DIAL – Distributed Interactive Analysis of Large Datasets• BNL project to study interactive analysis of large amounts of data
• Delivered framework is also suitable for non-interactive jobs
AJDL – Abstract Job Definition Language• Generic and extensible definition of jobs intended for use with any
batch or workload management system
• Components: transformation (application and task), dataset and job
![Page 4: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/4.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 4
David Adams
ATLAS
ADA/DIAL modelUser interaction scenario:
• Select a dataset– AJDL, not HEPCAL
• Examine dataset– Display content (event ID’s, type-keys)– Location of data (logical files, physical files, DB, …)
> Including means to localize data
• Define transformation– Application that processes the data– Task carrying data to configure application– Select these from catalog and modify
• Create job to apply transformation to dataset– Submit app, task and dataset to scheduler (e.g. analysis service)– Result will be a new dataset
![Page 5: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/5.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 5
David Adams
ATLAS
ADA/DIAL model (cont)User interactions (cont)
• Monitor job– Status (running, done, failed, …)
– Status of sub-jobs
– Examine partial result
• Examine final result (output dataset)
• Repeat
Following block diagram shows major components• Client
• High-level services
• Middleware services
![Page 6: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/6.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 6
David Adams
ATLAS
ADA/DIAL model (cont)
M id d lew ar e s er v ic e in te r f ac es
C EW M S F ileC ata lo g
S E . . .e tc . M id d lew ar es er v ic es
H ig h lev el s erv ice in terfaces (A J D L )
In te ra c t iv eA n a ly s isS e rv ic e
Batc hAn aly s isS er v ic e
AT P R O DAn aly s isS er v ic e
R O O Tc m d lin e
C lien t
G AN G Ac m d lin e
C lien t
G AN G AT as k
M an ag em en t
C ap o n eAn aly s isS er v ic e
G r ap h ic a lJ o b
Bu ild er
G AN G AJ o b
M an ag em en t
Hig h - lev e ls er v ic es
C lien t to o ls
AR D AAn aly s isS er v ic eC ata lo g
s er v ic es
G AN G A G UI
D atas e tS p lit te r
D atas e tM er g er
J o bM an ag em en t
Release 1.0 Started
![Page 7: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/7.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 7
David Adams
ATLAS
Current status of ADAFramework based on DIAL 0.90
Clients• ROOT from DIAL (using rootcint)
• Python from GANGA (using lcgdict with DIAL classes)
• GUI being developed by GANGA
![Page 8: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/8.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 8
David Adams
ATLAS
Current status of ADA (cont)Analysis services
• DIAL provides the starting point for all the following
• Interactive service (i.e. low latency for short jobs)– Currently uses a fast LSF queue
– Condor COD being tested
– Running at BNL
– Will be deployed at other sites
– May also be deployed for long-running jobs
• Service to insert jobs in ATLAS production queue is being developed
– Jobs may be run on any of the ATLAS grids
• ARDA service based on gLite being developed– Jobs sent to gLite WMS
![Page 9: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/9.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 9
David Adams
ATLAS
Current status of ADA (cont)Applications
• DC1 CBNT histograms in place
• DC2 reconstruction for release 8.0.1 in place
• Working on DC2 event data histograms
• Capability to reuse production xforms being developed
Datasets• Generic classes in DIAL
• DC1: CbntDataset and HbookDataset
• DC2: AtlasPoolEventDataset and RootHistogramDataset
Catalogs• AMI adding required tables and user interface
– Repositories, selection catalogs, …
• Programmatic access via AMI web service
![Page 10: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/10.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 10
David Adams
ATLAS
Connections to PPDGDIAL
• Provides many parts of ADA– The overall framework
– The ROOT client
– Infrastructure for all analysis services delivered and under development
– Connection to LSF, Condor (including COD)
• Depends on PPDG for support and exchange of ideas
![Page 11: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/11.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 11
David Adams
ATLAS
Connections to PPDG (cont)AJDL
• Facilitates the development of clients or services without DIAL
– E.g. plan for GANGA python interface is to drop dependence on DIAL C++ libraries
> Lightweight client for any platform
• Goal to standardize within PPDG– Send non-ATLAS jobs to ATLAS analysis service
– And vice versa
![Page 12: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/12.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 12
David Adams
ATLAS
Connections to PPDG (cont)U.S. grid infrastructure
• U.S. sites important to ATLAS – And especially to USATLAS
• Assume continuing support from PPDG– Grid3, grid3+, OSG, …
• Most sites continue taking jobs from ATPROD queues
• A few sites to also host interactive analysis service
![Page 13: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/13.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 13
David Adams
ATLAS
Connections to PPDG (cont)Monitoring and modeling
• US monitoring data fed into ATLAS computing model– Job data from ADA catalogs
– Site data from grid monitoring
• ACM is interested in constructing models to understand analysis patterns
– Collaboration with CS-11?
![Page 14: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/14.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 14
David Adams
ATLAS
Connections to PPDG (cont)EGEE middleware
• Install gLite on one or more U.S. sites– At least services required to support WMS
> CE, SE, …
• Submit jobs from the ATLAS-ARDA prototype
• Common project– Also submit jobs from CMS and Alice prototypes
• Will U.S. sites deploy EGEE middleware?– Important issue for evolution of ADA
![Page 15: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/15.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 15
David Adams
ATLAS
Connections to PPDG (cont)File cataloging and movement
• SRM likely to become standard– Licensing issues?
• Many replica catalogs relevant to ATLAS– Magda (phased out for DC2)– EDG RLS (LCG catalog)– ISI RLS (separate catalogs for U.S. and NorduGrid)– EGEE/AliEn file catalog (required for ARDA prototype?)
• Also many options for high-level interfaces– ATLAS internal system Don Quijote– CMS file transfer system– EGEE file transfer system (same as CMS?)– LBL RMS/RRS
• PPDG can help sort this out
![Page 16: David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin.](https://reader036.fdocuments.us/reader036/viewer/2022072017/56649f075503460f94c1cb0f/html5/thumbnails/16.jpg)
ATLAS and ARDA ARDA Workshop June 21, 2004 16
David Adams
ATLAS
ConclusionsATLAS has benefited in may ways from PPDG
• Magda, DIAL, AJDL, SRM, Grid3, …
• Ideas in HEPCAL and ARDA
Continue to look to PPDG• Forum to exchange ideas
• Provider or identifier of useful tools and services
• Support for U.S. grid infrastructure