The ATLAS Database Project Richard Hawkings, CERN Torre Wenaus, BNL/CERN ATLAS plenary meeting June...
-
Upload
myra-lawson -
Category
Documents
-
view
221 -
download
0
Transcript of The ATLAS Database Project Richard Hawkings, CERN Torre Wenaus, BNL/CERN ATLAS plenary meeting June...
The ATLAS Database Project
Richard Hawkings, CERNRichard Hawkings, CERN
Torre Wenaus, BNL/CERNTorre Wenaus, BNL/CERN
ATLAS plenary meetingATLAS plenary meeting
June 24, 2004June 24, 2004
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 2
Outline
Mandate and scopeMandate and scope
Project definition processProject definition process
Organization and communicationOrganization and communication
Subproject surveySubproject survey
Concluding remarksConcluding remarks
The current draft plan can be found at: The current draft plan can be found at: http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/calibration/AtlasDBProjectPlan.{ps,http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/calibration/AtlasDBProjectPlan.{ps,
pdfpdf
}}
(temporary location until new database web is up and running)(temporary location until new database web is up and running)
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 3
Project Mandate and Scope
Lead and coordinate all ATLAS database activitiesLead and coordinate all ATLAS database activities Including those so far under Software, TC, TDAQ, Detector Projects
Software, servers, distributed data management infrastructure
Specifically, databases and data management forSpecifically, databases and data management for Detector production, detector installation, survey data
Detector geometry
Online configuration, run bookkeeping, run conditions
Event data and metadata
Calibration and alignment (online and offline)
Offline processing configuration and bookkeeping
Grid based access to event and non-event data
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 4
Project Definition Process
RH, TW appointments (each currently at 50%) effective May 1RH, TW appointments (each currently at 50%) effective May 1stst 2004 2004
Serious project definition work began late AprilSerious project definition work began late April Based on the project outline presented at January 2004 software week Project should strengthen, not weaken or delay, DB activities across ATLAS
Many individual discussions, feeding into plan iterationsMany individual discussions, feeding into plan iterations
Public draft circulated to ATLAS software community in advance of BNL Public draft circulated to ATLAS software community in advance of BNL
software week, and discussed during the weeksoftware week, and discussed during the week No major new input – generally favourable impression
Draft plan was approved by May 28Draft plan was approved by May 28thth CMB/SPMB CMB/SPMB
Continuing to gather input (TDAQ community, subdetectors, EB, …)Continuing to gather input (TDAQ community, subdetectors, EB, …)
The plan will evolve, but the current version will guide project launchThe plan will evolve, but the current version will guide project launch
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 5
Organization
Part of Software and Computing Project (Dario Barberis) Part of Software and Computing Project (Dario Barberis)
Richard Hawkings, Torre Wenaus Co-Leaders 5/04-5/06Richard Hawkings, Torre Wenaus Co-Leaders 5/04-5/06 Computing Management Board (CMB) members Ex-officio Trigger/DAQ steering group (TDAQ-SG) members
Database Steering Group chaired by the Project Leaders is Database Steering Group chaired by the Project Leaders is
planning and decision making bodyplanning and decision making body
Twelve subprojects cover the mandated scopeTwelve subprojects cover the mandated scope
Some subprojects embedded in other parts of ATLASSome subprojects embedded in other parts of ATLAS Where tight integration of DB activities must be preserved Subprojects being organized in close consultation with the projects
concerned, to ensure this happens
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 6
Work Breakdown
1)1) Project managementProject management - steering, planning, coordination, strategy - steering, planning, coordination, strategy
2)2) Detector productionDetector production - long term storage of subdetector production data - long term storage of subdetector production data
3)3) Detector installationDetector installation - manufacturing and test (MTF), racks, cabling, survey - manufacturing and test (MTF), racks, cabling, survey
4)4) Detector geometryDetector geometry - primary numbers for detector description - primary numbers for detector description
5)5) Online databasesOnline databases - configuration, conditions, bookkeeping, offline transfer - configuration, conditions, bookkeeping, offline transfer
6)6) Calibration and alignment Calibration and alignment - central tools, not subdetector algorithm work- central tools, not subdetector algorithm work
7)7) Conditions database infrastructureConditions database infrastructure - core sw and tools; framework integration - core sw and tools; framework integration
8)8) Event dataEvent data - events and metadata from raw to analysis. Common core sw - events and metadata from raw to analysis. Common core sw
9)9) Distributed data managementDistributed data management - event and conditions data. Grid integration - event and conditions data. Grid integration
10)10) Offline processing configuration and bookkeepingOffline processing configuration and bookkeeping - production metadata - production metadata
11)11) Distributed database servicesDistributed database services - physical databases, distributed infrastructure - physical databases, distributed infrastructure
12)12) Software support servicesSoftware support services - supporting users and deployers of DB software - supporting users and deployers of DB software
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 7
Steering Group
Planning and decision making bodyPlanning and decision making body
Integration mechanism to ensure synergy and coherenceIntegration mechanism to ensure synergy and coherence Across subdetectors, across database areas
Representation from subprojects, associated projectsRepresentation from subprojects, associated projects
Decisions taken following consensus in the steering groupDecisions taken following consensus in the steering group
Project Leaders have full authority for planning and execution, including Project Leaders have full authority for planning and execution, including
cases lacking full consensuscases lacking full consensus In cases of serious dissent, CMB and TDAQ-SG (where appropriate) take
final decision
Strategic decisions go to the CMB (and TDAQ-SG) for endorsementStrategic decisions go to the CMB (and TDAQ-SG) for endorsement
Steering Group is a large body, as the broad scope requires…Steering Group is a large body, as the broad scope requires…
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 8
Steering Group Composition
Still have to fill some appointments in consultation with appropriate communitiesStill have to fill some appointments in consultation with appropriate communities Technical coordination - Kathy Pommes, Luc Poggioli Online - Antonio Amorim, Mihai Caprini, Igor Soloviev High level trigger - TBD Calibration and Alignment - Richard Hawkings Detector geometry - Joe Boudreau Inner Detector - TBD LAr calorimeter - Hong Ma Tile calorimeter – Karl Gellerstedt Muon spectrometer - Joe Rothberg Conditions database infrastructure - RD Schaffer Event data - David Malon Distributed data management - TBD Offline processing - TBD Distributed database services - Alexandre Vaniachine User feedback - TBD, an informed+noisy+constructive user voice Persistency Framework Project (LCG Apps Area) - Dirk Duellmann Computing Coordinator (ex officio) - Dario Barberis Software Project Leader (ex officio) - David Quarrie
Liaison from DB Project to
Software Project Management Board - David Malon
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 9
Communication
MeetingsMeetings All with agendas in advance, and minutes documenting technical progress,
planning and decisions. Phone connections to allow wide participation Steering Group meeting bi-weekly (Friday 15:30 starting June 25th) Weekly meeting covering primarily offline – continuation of existing meeting
Technical planning and execution, within overall guidelines and plan of the SG
A second weekly slot (to be defined) for Online database meeting, roughly bi-weekly TC database meeting (production, installation) every 2-4 weeks Conditions data working group meeting periodically Associated mailing lists for all these communities (online/TC to be setup)
WebWeb Project web as a comprehensive and current source of technical and
planning information and documentation is a very high priority We take this as a project management responsibility We will both write web content and nag others to do the same!
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 10
Subproject Survey
A compressed survey of the subprojects…A compressed survey of the subprojects…
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 11
1) Project Management
Most of this area already addressed…Most of this area already addressed… Planning and steering
Project meetings
Project web
……but also includes…but also includes… Monitoring of QA, testing and validation
Strategy and technology evolution
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 12
2) Detector Production
The many production/construction DBs used worldwide in the The many production/construction DBs used worldwide in the
subdetectors are not the responsibility of this projectsubdetectors are not the responsibility of this project
Ensuring Ensuring allall data of long-term interest to ATLAS is gathered into central data of long-term interest to ATLAS is gathered into central
(CERN IT Oracle) databases (CERN IT Oracle) databases isis in the mandate in the mandate Central system for uniform access and long-term maintainability
Provision of tools, standards, guidelines to subdetectors
Data definition and entry is subdetector responsibility
Central DB exists and some subsystems are entering data, but there is Central DB exists and some subsystems are entering data, but there is
a great deal of central/common work to be donea great deal of central/common work to be done
Personnel and oversight for central/common work is mostly absentPersonnel and oversight for central/common work is mostly absent Some ideas and possibilities – benefit from subdet production finishing?
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 13
3) Detector Installation
MTF (manufacturing and test) installation databaseMTF (manufacturing and test) installation database Installed parts with links to production database
Rack database (exists; being populated)Rack database (exists; being populated)
Cabling database (partially exists)Cabling database (partially exists)
Survey database (does not exist)Survey database (does not exist)
Extraction tools to e.g. use cabling data for online config Extraction tools to e.g. use cabling data for online config
(not existing, and needed soon – e.g. comissioning)(not existing, and needed soon – e.g. comissioning)
Here again, personnel and oversight for central/common Here again, personnel and oversight for central/common
work (this project’s mandate) is severely lackingwork (this project’s mandate) is severely lacking
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 14
4) Detector Geometry
Primary numbers used by detector description softwarePrimary numbers used by detector description software
NOVA-based system deployed and operating for some timeNOVA-based system deployed and operating for some time
Work is underway to move to a successor with versioning Work is underway to move to a successor with versioning
supportsupport Approach is consistent with EB-mandated push to implement final
‘as-built’ detector geometry before subdetector engineers leave
Involves a ‘fast track’ implementation using standard relational DB
tools to quickly support gathering and loading data
Offline access (via LCG ‘relational POOL’, conditions DB) on longer
timescale when that software is ready
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 15
5) Online Databases
Configuration database - 30% of the online system; must Configuration database - 30% of the online system; must
remain integral to onlineremain integral to online
Online run bookkeeping - expect to employ standard Online run bookkeeping - expect to employ standard
offline/online tools offline/online tools
Conditions database interfaces - ditto. Conditions database interfaces - ditto. Online, through the Lisbon group, has provided the standard tool
used also offline, now also contributing to LCG CondDB project
External interfaces and data flow - Information Server (IS), External interfaces and data flow - Information Server (IS),
slow controls (DCS), offline (to AMI, production mgmt)slow controls (DCS), offline (to AMI, production mgmt)
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 16
6) Calibration and Alignment
Activities organized via conditions data working groupActivities organized via conditions data working group
Communication forum for developing strategies, preparing Communication forum for developing strategies, preparing
online and offline algorithmsonline and offline algorithms
Conditions database loading and accessConditions database loading and access
Contribute to computing model Contribute to computing model Little manpower for central/common tasks
Both subdetector and coordination effort currently focused on CTB
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 17
7) Conditions Database Infrastructure
Conditions database core software developmentConditions database core software development
Supporting tools (browsers, data distribution and synchronization, Supporting tools (browsers, data distribution and synchronization,
subsetting, etc.)subsetting, etc.)
Athena services for conditions dataAthena services for conditions data
ATLAS participation in LCG CondDB common projectATLAS participation in LCG CondDB common project Activity is increasing, with ATLAS the largest experiment participant New: Relational DB support for POOL, versioning system component
Short-term focus is on CTB support and stabilityShort-term focus is on CTB support and stability
Planning (after CTB) to converge from many tools …Planning (after CTB) to converge from many tools … Lisbon CondDB, NOVA, POOL, geometry DB
… … to essentially one, incorporating all experience gainedto essentially one, incorporating all experience gained Common project CondDB with POOL support
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 18
8) Event Data
Core software support for event data, from raw data to analysisCore software support for event data, from raw data to analysis Including event collections and physics datasets
Athena integration - both event data specific and common persistency Athena integration - both event data specific and common persistency
services services This activity moved from Software Project to Database Project
Event data access outside Athena, e.g. in ROOT analysis environmentEvent data access outside Athena, e.g. in ROOT analysis environment
ATLAS participation in POOL common projectATLAS participation in POOL common project
Event data storage for CTB and DC2 is generally OK
File-level data management is handled by the next subproject
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 19
9) Distributed Data Management
Management of ATLAS data around the worldManagement of ATLAS data around the world Cataloging, replication, synchronization, access control, …
Event, conditions and other data; files and relational DBsEvent, conditions and other data; files and relational DBs
Integration/interfacing with grid tools for data managementIntegration/interfacing with grid tools for data management And working around grid software deficiencies
Present focus on DC2 production needs – Present focus on DC2 production needs – Key tool: Don Quixote – interface to heterogeneous grids
As yet no overall strategy for DDM today and in the futureAs yet no overall strategy for DDM today and in the future Need urgently to address user-level data management tools
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 20
10) Offline Processing Configuration and Bookkeeping
Databases cataloging metadata that is input to and output Databases cataloging metadata that is input to and output
from offline processing jobsfrom offline processing jobs Both managed production and (in the future) group and individual
level jobs
Cataloging of provenance information to unambiguously define
job/software configuration
Key tools at present are AMI and the production DBKey tools at present are AMI and the production DB
Again needs plan & strategy, including technology choicesAgain needs plan & strategy, including technology choices
Present focus on DC2 and CTB supportPresent focus on DC2 and CTB support
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 21
11) Distributed Database Services
Support for deployed database and data management Support for deployed database and data management
services at CERN and throughout ATLASservices at CERN and throughout ATLAS
Physical servers, distributed (heterogeneous!) database Physical servers, distributed (heterogeneous!) database
infrastructureinfrastructure
Support and/or liaison for admin and operations of Support and/or liaison for admin and operations of
databases away from CERNdatabases away from CERN
Liaison to CERN IT/DB for CERN-based servicesLiaison to CERN IT/DB for CERN-based services Possible common project in distributed database infrastructure
under discussion, initiated by ATLAS (David Malon) Present focus is again on CTB and DC2 support
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 22
12) Database Software Support Services
Support for software, distinct from support for physical Support for software, distinct from support for physical
services (preceding subproject)services (preceding subproject)
DocumentationDocumentation Not authoring (developers are responsible), but organization,
usability, monitoring and review, ‘encouragement’ to authors
Tutorials and trainingTutorials and training
User support servicesUser support services E.g. Savannah problem reporting, feature requests
June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 23
The challenges ahead
ATLAS database project is a big projectATLAS database project is a big project Covers many different areas, diverse communities
Key objectives: Improving communication, facilitating data transfers
Short and medium-term concernsShort and medium-term concerns Manpower for TC-related areas (detector production / installation)
Missing both sub-project leadership effort and workers
Becoming increasingly important as we approach commissioning Can we exploit effort freed up from sub-detectors ?
Data management strategies and needs – DC2 and vision beyond Large scale distributed infrastructure – LCG common project initiative Individual doing analysis/development – end user tools
New contributions / efforts are needed and welcome !New contributions / efforts are needed and welcome !