Organization and Management of ATLAS Nightly Builds F. Luehring a, E. Obreshkov b, D.Quarrie c, G....
-
Upload
herbert-mathews -
Category
Documents
-
view
214 -
download
0
Transcript of Organization and Management of ATLAS Nightly Builds F. Luehring a, E. Obreshkov b, D.Quarrie c, G....
Organization and Management of ATLAS Nightly Builds F. Luehringa, E. Obreshkovb, D.Quarriec, G. Rybkined, A. Undruse
University of Indiana, USAa, DESY, Germanyb , LBNL, USAc ,LAL, Franced ,BNL, USAe
Organization and Management of ATLAS Nightly Builds F. Luehringa, E. Obreshkovb, D.Quarriec, G. Rybkined, A. Undruse
University of Indiana, USAa, DESY, Germanyb , LBNL, USAc ,LAL, Franced ,BNL, USAe
Abstract
25 branches of automated multi-platform software nightly builds is a major component in ATLAS collaborative software organization and code approval scheme. The successful nightly releases are transformed into stable releases used for data processing worldwide. ATLAS nightly builds are managed by the fully automated NICOS framework [1]. The modular structure of NICOS allows for an easy integration of third-party build and validation tools. The ATN [2] test tool is embedded within the nightly system and provides the first results even before the full compilations completion. Several ATLAS test frameworks are synchronized with NICOS jobs and run larger production jobs with the nightly releases. NICOS web pages dynamically provide information about the progress and results of the builds. For faster feedback the e-mail notifications about nightly build problems are automatically distributed to responsible developers.
Abstract
25 branches of automated multi-platform software nightly builds is a major component in ATLAS collaborative software organization and code approval scheme. The successful nightly releases are transformed into stable releases used for data processing worldwide. ATLAS nightly builds are managed by the fully automated NICOS framework [1]. The modular structure of NICOS allows for an easy integration of third-party build and validation tools. The ATN [2] test tool is embedded within the nightly system and provides the first results even before the full compilations completion. Several ATLAS test frameworks are synchronized with NICOS jobs and run larger production jobs with the nightly releases. NICOS web pages dynamically provide information about the progress and results of the builds. For faster feedback the e-mail notifications about nightly build problems are automatically distributed to responsible developers.
Nightly FrameworkNightly Framework
References[1] A. Undrus, CHEP 03, La Jolla, USA, 2003, eConf C0303241, TUJT006 [hep-ex/0305087]; http://www.usatlas.bnl.gov/computing/software/nicos/index.html
[2] A. Undrus, CHEP 04, Interlaken, 2004, Conference proceedings, p. 521
[3] http://www.cmtsite.org
[4] Tag Collector reference
[5] AMI reference
[6] PackDist reference
[7] RTT reference
[8] FCT reference
[9] TCT reference
[10] KV rererence
References[1] A. Undrus, CHEP 03, La Jolla, USA, 2003, eConf C0303241, TUJT006 [hep-ex/0305087]; http://www.usatlas.bnl.gov/computing/software/nicos/index.html
[2] A. Undrus, CHEP 04, Interlaken, 2004, Conference proceedings, p. 521
[3] http://www.cmtsite.org
[4] Tag Collector reference
[5] AMI reference
[6] PackDist reference
[7] RTT reference
[8] FCT reference
[9] TCT reference
[10] KV rererence
NICOS FRAMEWORK
NIGHTLY CONFIGURATION
DISTRIBUTION
CODEMANAGEMENT
TESTING
nightlyrelease
CMT build
ATN testing
AMI DB
Tag Collector
CVSRepository
RTT
FCT
TCT
KITVALIDATION
KITSCERN AFSATLAS
TierCenters
NICOSWEB PAGES
NICOS is connected with ATLAS collaborative tools:CMT [3] code management and build toolTag Collector [4] web based tool for managing the tags of packages in releaseATLAS CVS code repositoryATLAS metadata DB (AMI) [5] for storage of NICOS configurations for different nightlies branchesNightly releases are installed for worldwide access at CERN AFS and also can be downloaded with ATLAS distribution kits tools [6]Integrated ATN [2] and external RTT [7], FCT [8], TCT [9], Kit Validation [10] testing frameworks
NICOS Web PagesNICOS Web Pages
All nightlies summary with message of day, distribution kit and tests status
Nightlies computer farm statuswith dynamic information on nodes
Log files with error diagnostics with links to problem messages
Nightly branch summary with results for inside projects and platforms
Summary for project/platformwith latest releases comparisons
Nightly release summary with diagnostics for individual packages
Software Release StructureSoftware Release Structure
AtlasOffline
ATLAS PROJECTS
ATLAS PATCH PROJECTS
PACKAGESa) Containers: directories for related packagesb) Leaf: source code and/or scriptsc) Glue: interfaces to externals ……….Number of leaf + container packages:
EXTERNALS
AtlasAnalysis
AtlasTrigger
AtlasReco
AtlasSimulation
AtlasEvent
AtlasConditions
AtlasCore
GAUDI Framework
DetCommon
Data QualityMonitoring
Data AcquisitionSoftware
Interfacesto externals(“LCGCMT”)
Ext
erna
ls (
RO
OT
, G
eant
, …
)
26+20
234+84
187+30
452+117
190+35
293+90
206+55
178+20
15+5
Total: 1781 leaf,456 container
packages as of 02/15/09
Projects are groups of packages with similar dependencies managed by CMT [3]. They are built as units and can evolve at different rates. Patch projects sit at the top of project hierarchy. They contain override versions of packages from downstream projects.
Software ValidationSoftware Validation
Release Coordinatorensure design goals
Chief Architect
Validation Data Managerprovides data samples
Production Manageroversees jobs definitions
Project Coordinators: evaluate new software submissions
Package Managers: request upgrades through Tag Collector
New software versions are verified in validation or migration nightlies. Upon approval from coordinators they are moved to the primary branch. Stable software releases are created from successful primaries. All patches to stable releases are sent to primary branch. Package versions associated with releases and project dependencies are handled by the Tag Collector.
Primary development nightly branch
Validation nightly branch
Migration nightlies:Verification of disruptive changes
LCG nightlies:Verification of externals upgrades
Stablerelease
Stablerelease
Stablerelease
Patchrelease
Patchrelease
ATLAS Nightlies in Numbers
Number of branches: 25 Total number of platforms in all branches: 40 All nightlies are rebuilt and tested daily Old nightly releases are kept for 2 - 7 days Nightlies computing farm:• 40 nodes, 4x3.0 or 8x2.33 GHz• 8 - 16 GB RAM Time to rebuild nighty release: 10 hours Number of ATN tests: 300 Additional time to complete ATN tests: 5 hours
ATLAS Nightlies in Numbers
Number of branches: 25 Total number of platforms in all branches: 40 All nightlies are rebuilt and tested daily Old nightly releases are kept for 2 - 7 days Nightlies computing farm:• 40 nodes, 4x3.0 or 8x2.33 GHz• 8 - 16 GB RAM Time to rebuild nighty release: 10 hours Number of ATN tests: 300 Additional time to complete ATN tests: 5 hours
NICOS DesignNICOS nightly jobs consist of tightly synchronized streams. The results for different platforms are combined in a single release.
NICOS DesignNICOS nightly jobs consist of tightly synchronized streams. The results for different platforms are combined in a single release.
Tag Collector access Code checkout proj.1 Code checkout proj.2 Code checkout proj.3
Build project 1 Build project 2 Build project 3
Install 2Install 1
Tests project 1 Tests project 2
Build project 1 Build project 2 Build project 3
Install 1 Install 2wait