Post on 17-Jan-2016
Metadata and COMA
Elizabeth Gallas
Oxford.
ATLAS Weekly / Open EB MeetingOctober 18, 2011
Outline Overview of ATLAS Metadata Interfaces What is COMA ? Principles Tour
VERY quick tour today…hope you will try yourself! Summary and Plans Conclusions
Backup Slides: Links
Thanks to many experts, developers and also many of you who post questions / use cases to e-groups !!!
June 2011 Elizabeth Gallas - COMA 2
Subsystem specific metadata applications... driven by use cases/workflow Trigger: TriggerTool … Web: trigconf and trigger timelines Geometry DB: Detector Description Browser Conditions DB:
runQuery (Run information from Conditions DB) ATLAS WEB DQ COOL Tag Browser
Data Summary Reports (Luminosity, Beam) Beam Spot Summary GANGA and PAthena Panda / monitor DQ2 Client Tag Collector – software releases ...
Dedicated Metadata Catalogs TAGs – event level (file and relational formats)
iELSSI and Suite of TAG Services TAG Catalog AMI – Dataset, processing … other metadata
And the AMI Suite of services COMA – Conditions and configuration
ATLAS Metadata Application Overview
June 2011 Elizabeth Gallas - COMA 3
Connected in many ways (and with links to many above applications) …Some mentioned today in the course of “COMA” topic discussed today
Medicine: deep sleep … a state of extreme unresponsiveness
? coma ? Many definitions:
June 2011 Elizabeth Gallas - COMA 4
Optics: aberration inherent to certain optical designs or due to imperfection in the lens or other components…
Astronomy: luminous cloud surrounding the frozen solid nucleus in the head of a comet …
ATLAS: COMA is an ~ acronym for a metadata catalog
and its related services:
Conditions / Configuration Metadata for ATLAS
June 2011 Elizabeth Gallas - COMA 5
What is COMA ? COMA: “Conditions/Configuration Metadata for ATLAS”
Originally: built to support dynamic queries of TAG DB Evolved: into a standalone system with interfaces Now part of general effort to consolidate/relate ATLAS Metadata
COMA Components: Relational Database
Contains extracted, refined, reduced, and derived information from system specific data sources (e.g. Conditions DB, …) plus information from non-database sources.
Interfaces A set of Browsers and Reports
Main Documentationhttps://twiki.cern.ch/twiki/bin/viewauth/Atlas/ConditionsMetadata
COMA Portal (grid certificate in browser is required)https://atlas-tagservices.cern.ch/tagservices/RunBrowser/index.html
June 2011 Elizabeth Gallas - COMA 6
COMA Interfaces: General Principles Report or Browse the COMA catalogue:
Originally developed to check COMA loading and do cross checks Evolved to current form: on request from experts and users
Logical division of information into separate but cross linked reports Include links to more details (COMA links and External links)
runQuery, Luminosity, AMI, Trigger, DQ …
Provide reports [distinctive from, complimentary to] other systems (Avoid duplication of effort) Display COMA ‘derived’ information
Not available in other systems
Note: All the COMA Reports use GET methods: Inputs are not complex, user can modify URL to get report they want,
put links into TWiki or other documentation
https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=161562
June 2011 Elizabeth Gallas - COMA 7
COMA Interfaces Portalhttps://atlas-tagservices.cern.ch/tagservices/RunBrowser/index.html
Tour
stops
today
DB mtgWeds
June 2011 Elizabeth Gallas - COMA 8
Introduction: ATLAS Data Periods A Data Period is a set of ATLAS Runs grouped for a purpose
Defined by Data Preparation Coordinators Used in ATLAS data processing, assessment, and selection … Each Period uniquely defined with a combination of
Project name (i.e. ‘data10_7TeV’) Period name (i.e. ‘C1’, ‘C2’, ‘C’, ‘AllYear’ …)
Before 2011, Data Periods were Described on TWiki page
https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods Stored in a file based system
Edited by hand by Data Prep Coordination (experts) Structure evolved over last year with experience
This experience valuable to decide/define long term solution Early in 2011: Data Periods stored in the COMA DB
Thanks: Beate (DataPrep Coordinator), AMI team, me.
June 2011 Elizabeth Gallas - COMA 9https://atlas-tagservices.cern.ch/RBR/rBR_Period_Report.php
Menu Purpose: Show DataPrep defined Periods giving links to reports of general info about their Runs.
Choose the Period of interest: By Year
E.G. all ‘2010’ By Project
e.g. ‘data10_7TeV’ By specific Period or Group
Click on the project and then Period of interest
Wildcards can be entered in this optional section, then click on Submit button
“highlight”expanding
sections
June 2011 Elizabeth Gallas - COMA 10
Period Description Report Example for data10_7TeV
-/+ highlighted links:
These sections expand
to show period members
Members of
data11_7TeV.M
are M1, M2, M3, M4
Links: to COMA & runQuery
multi-Run Reports
for chosen Period
(COMA link: next slide)
Header:
Input criteria;
Links: Contact,
Documentation
Period “Status”: M1 is ‘frozen’,
M2-M4 are ‘locked’ … M ‘unlocked’ (M5 is on-going)
June 2011 Elizabeth Gallas - COMA 11
COMA Multi-Run Report:Purpose: show basic information for all runs matching the input criteria
Run links: COMA, runQuery, and
LumiDataSummary Single Run report
(COMA link: next slide)
SMK links:
COMA Trigger Super Master Key Report
Links in Table column headers:
Short description of column
Hover over any link:
Indicates what
will happen
https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?fnt=data10_7TeV&pn=VdM
June 2011 Elizabeth Gallas - COMA 12
COMA Single Run 180776 Overviewhttps://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=180776
General Run info: Project, LB range, Date … Events, HLT Release … Links to COMA reports
COMA Master Key report (no time to show today)
Links to external reports HLT release, AMI, LumiDS,
runQuery, Trigger, PSEvol
Yellow sections expand: Data Periods Active Streams AMI tags (TAG production)
Shortcut find runs w/AMI tag Prescale Evolution
PS Key changes Trigger section
All HLT signaturesAll LVL1 items
June 2011 Elizabeth Gallas - COMA 13
COMA Single Run -/+AMITag SectionRecent data11_7TeV M4 Period Run 190933
Shows: All physics streams
Including debug ! “First Pass” – first processing
New columns appear for reprocessing passes
AMI tag used to process data Link: COMA multi-Run Report
All runs processed w/this tag “?”: link to AMI
Description of AMI tag
Launch iELSSI ! For this Run/Stream/AMItag Take a quick look at this run with
the ELSSI TAG browser !!! See if events pass your trigger in
the debug stream …
June 2011 Elizabeth Gallas - COMA 14
COMA Single Run -/+Trigger SectionStart to see here some COMA “derived” info
Open this section in next slide…
COMA divides HLT trigger signatures: “Physics”: complete EF-L2-L1 chains “Commissioning”: has missing level(s)COMA looks at each LB range:1. Decides if each chain is “active” based
on prescales, passthrough, rerun in every LB range of Prescale evolution
2. For “active” chains via prescale, calculates an “Aggregate prescale” (EF*L2*L1 prescale) in each LB range
3. Determines Min, Max of AggPS over the Run for each chain
4. Determines Run and LB-wise PS,PT,RR Flags
June 2011 Elizabeth Gallas - COMA 15
COMA single-Run Report: “Physics” Trigger SectionChoose (using jQueryColumnFilters)
Chains: EF*JPSI and
Stream: egamma
COMA Chain Report:
All Runs w/ this chain/item
COMA Prescale Report:
LB-wise PS info for
this Run/Signature
(next slide)
Run-wise Aggregate prescale Range
and Flags(PT,PS,RR)
June 2011 Elizabeth Gallas - COMA 16
COMA Prescale Report (Run 180776,EF_e9_tight_e5_tight_Jpsi)
LB Range Summary table: LB-wise Prescale Evolution Link:Trigger DB Prescale
Reports (useful cross check !) Prescales:
L1,L2,EF and Aggregate PS Chain Inactive from
LB 1-99 and 114-134(LB ranges in COMA always inclusive)
Next slide:COMA Chain Report forEF_e9_tight_e5_tight_Jpsi
Run-wise Summary: Includes links to COMA reports:
Run, Master Key, Chain and Prescale Reports (each level)
Associated L2, L1 names Aggregate PS Range: 4 to 5 Flags:
Partially active via PS Never passthrough (L2 or EF) Never Rerun
June 2011 Elizabeth Gallas - COMA 17
RBR Chain Report (EF_e9_tight_e5_tight_Jpsi)
Activation Summary:Shows the Runs in which this element is
“active” separated by: “Physics” EF-L2-L1 signatures
Via prescale Via pass through Via rerun
“Commissioning” signatures Via prescale Via pass through Via rerun
Configuration Summary:Shows where this element is configured:
Super Master Key(s), Runs (Summary)
Period Evolution: Shows chain/item bit, version evolution
for “Physics” chains during Period Runs
Open these sections in next slides…
June 2011 Elizabeth Gallas - COMA 18
COMA Chain Report (EF_e9_tight_e5_tight_Jpsi)Expand Run-wise Activation … “Physics” EF-L2-L1 signatures
Active via Prescale Runs in Data Periods
Table Shows (Run Count): Periods, Link: Run, SMK Reports Level bit assignments Link to: Chain/Item Reports (3) Range of Aggregate Prescale while
chain is active via prescale in Run Links: COMA Prescale Report (3)
70 Period Runs where this chain is “active”
June 2011 Elizabeth Gallas - COMA 19
COMA Chain Report (EF_e9_tight_e5_tight_Jpsi)Period Evolution Section … For “Physics” chains in Period Runs Separate table for each EF-L2-L1
signature at each beam energy
Each Table Shows: Row-wise:Distinct set of bit and
chain/item versions
Columns: Bit assignments Chain version (links to Trig diff) Chain Report Links Range of AggPS, SMK, Run,
Period, Date, HLTRelease
Thanks to Tomasz, Joerg
for many useful discussions
June 2011 Elizabeth Gallas - COMA 20
COMA Chain Wildcard Report (input: “EF_g10%”)Purpose: See all the names matching a pattern or
Find exact name from part of the nameReport: Displays chain/item names matching the input string …
text size proportional to occurrence in SMK In Period Runs and in All Runs
June 2011 Elizabeth Gallas - COMA 21
Summary and Plans COMA – an integral part of ATLAS Metadata infrastructure
Essential to ATLAS event-level metadata decoding Ideally placed to provide links and interface to other metadata
Special relationship to AMI (and TAG catalog) Launch iELSSI to take a quick look at any Run
Primary source for “ATLAS Data Periods” Periods in Lum, DQ, Run Summary, AMI reports comes from COMA
Reports feature “derived” information not available elsewhere Trigger experts recommend COMA Trigger/Prescale
Report usage: from ~200 to over 5000 pages viewed/month. Peaked in July as users did final preparations for summer conferences
Current efforts: COMA Database content growing
Watch use cases to identify new areas to focus growth COMA Report and Browser development
keep pace with content, improve functionality and usability Beyond COMA: Interface development
Connecting COMA to other parts of the infrastructure
June 2011 Elizabeth Gallas - COMA 22
COMA Conclusions
This is an evolving system … information in the system is growing based on information available and use cases Adding more dimensions to the Conditions data
With suitable relationships to facilitate queries Making that criteria available in dynamic useable interfaces
We want to insure the Metadata is complete enough to satisfy use cases while reflecting accurately its limitations
Interfaces are being constructed to use selection syntax, criteria, and communication in common use in ATLAS This facilitates cross checks with other systems
Continuous process: talking with various experts to ensure data integrity, completeness, compatibility w/other systems
… Very positive feedback so far … more always welcome …
hn-atlas-physicsMetadata@cern.ch
June 2011 Elizabeth Gallas - COMA 23
Backup
Link SummaryMain TWiki Documentation:https://twiki.cern.ch/twiki/bin/viewauth/Atlas/ConditionsMetadata
COMA Report Documentation:https://atlas-tagservices.cern.ch/RBR/rBR_Documentation.html
COMA Portal (grid certificate in browser is required)https://atlas-tagservices.cern.ch/tagservices/RunBrowser/index.html
Trigger General Meeting (June 8, 2011):“User Guide to Trigger/Prescale information in COMA Reports”https://indico.cern.ch/materialDisplay.py?contribId=0&materialId=slides&confId=125723Data sources, Business rules and Caveats; Form input; Example links; details of Data Period & derived quantities;
Data Periods TWiki (revised):https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods
June 2011 Elizabeth Gallas - COMA 24
June 2011 Elizabeth Gallas - COMA 25
COMA Report(s) Example LinksCOMA single-Run Report (Run 161562):https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=161562
General Run information, AMI tags, DQ, Prescale Evolution, Triggers Trigger tables show derived Run-integrated prescales, passthrough
COMA Trigger Super Master Key Report (Key 877):https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?smk=877
Runs using this SMK, physics/commissioning chains, Level 1 items
COMA Chain Report (Chain EF_g11_etcut):https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?cn=EF_g11_etcut
Summary: SMK, Runs, Signature appearances of this chain Runs with this chain ‘active’
COMA Chain Wildcard Report (chain name matching string EF_g1%):https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?cn=EF_g1%
Summary: SMK, Runs with this chain: Configured, Active
COMA Prescale Report (Run 162882 Prescales for chain EF_g11_etcut):https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=162882&cn=EF_g11_etcut
Derived chain activity over the Run and LB-wise prescales
June 2011 Elizabeth Gallas - COMA 26
Data Periods: Links to Reports and Services
Revised TWiki page:https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods
Interactive USERS COMA Data Period Documentation Interface https://atlas-tagservices.cern.ch/RBR/rBR_Period_Report.php
Comments: hn-atlas-physicsMetadata@cern.ch.
Programmatic USERSFor systems needing period info: runQuery, beamspot, Data Quality, …, “Data Period Services” provided via pyAMI: http://ami.in2p3.fr/opencms/opencms/AMI/www/Client/DataPeriods_pyAMI.pdf
Comments: AMI / Tag_Collector Team.
Data Preparation EXPERTS: Entry Interface: https://ami.in2p3.fr/AMI/servlet/net.hep.atlas.Database.Bookkeeping.AMI.Servlet.Command?linkId=1479
Comments: AMI / Tag_Collector Team.
June 2011 Elizabeth Gallas - COMA 27
COMA runBrowserReport Index page INDEX: https://atlas-tagservices.cern.ch/RBR/runBrowserReport.html GUIDE: https://atlas-tagservices.cern.ch/RBR/rBR_Documentation.html
“Index” page: Enter one/more input quantity,
IE: a Run number IE: a Chain Name
Click on submit (or return key)
Links at bottom of every page:
Reports use “GET” methods:Modify the URL to change input
Reports contain linksNo need to use index page for
every queryLinks lead you to
other COMA reports other external reports
AMI, runQuery, TriggerDB…