4 December 2003 Natalia Ratnikova Natalia Ratnikova 1 Software Packaging with DAR Natalia Ratnikova,...

4 December 20034 December 2003 Natalia Ratnikova <[email protected]>Natalia Ratnikova <[email protected]> 11

Software Packaging Software Packaging with DARwith DAR

Natalia Ratnikova, Anzar Afaq, Greg Graham Natalia Ratnikova, Anzar Afaq, Greg Graham

FermilabFermilab

Tony Wildish, Veronique LefebureTony Wildish, Veronique Lefebure

CERNCERN


ACAT 2003, KEK, Japan: Software Packaging with DARACAT 2003, KEK, Japan: Software Packaging with DAR

Introduction Introduction ► ► MotivationMotivation

CCompact ompact MMuon uon SSolenoid olenoid CMSCMS HEP experiment HEP experiment will run on the LHC accelerator at CERN. will run on the LHC accelerator at CERN.

CMS is using GRID technologies to utilize CMS is using GRID technologies to utilize available computing resources for the worldwide available computing resources for the worldwide distributed Monte Carlo event production.distributed Monte Carlo event production.

To make this possible CMS software applications To make this possible CMS software applications must be brought to the production sites. must be brought to the production sites.



Introduction Introduction ► ► ScopeScope

CMS software includes a wide range of inter-CMS software includes a wide range of inter-related projects and external tools managed by related projects and external tools managed by the the SSoftware oftware CConfiguration onfiguration RRelease elease aand nd MManagement tool anagement tool SCRAMSCRAM. .

Complete installation of the CMS software and Complete installation of the CMS software and environment on the remote sites is uneasy task, environment on the remote sites is uneasy task, and actually it is not necessarily required in and actually it is not necessarily required in order to run ready applications. order to run ready applications.



Introduction Introduction ► ► GoalGoal

The USCMS software and computing project goal was to The USCMS software and computing project goal was to move CMS MC production in the US completely onto the move CMS MC production in the US completely onto the GRID computing resources.GRID computing resources.

We wanted to have an automated way to create self-We wanted to have an automated way to create self-consistent distributions of the applications, based on the consistent distributions of the applications, based on the software released at CERN.software released at CERN.

The The DDistribution istribution AAfter fter RRelease elease DARDAR tool was developed tool was developed at Fermilab for at Fermilab for quick-an-easy deploymentquick-an-easy deployment of the CMS of the CMS software applications, which can run on the systems that software applications, which can run on the systems that do not have pre-existing CMS environment. do not have pre-existing CMS environment.



DAR conceptDAR concept

DAR automatically creates and installs softwareDAR automatically creates and installs software applicationsapplications based on the based on the runtime environmentruntime environment . .

ApplicationApplication is a complete, self-contained software is a complete, self-contained software program, including all required shared libraries and other program, including all required shared libraries and other files, that can be executed in a particular environment to files, that can be executed in a particular environment to accomplish a particular computing task.accomplish a particular computing task.

Runtime environmentRuntime environment is a set of UNIX shell is a set of UNIX shell environment variables used by the program during the environment variables used by the program during the runtime.runtime.



Concept Concept ► ► Choices, Decisions Choices, Decisions

There is a class of tools and utilities, such as operating There is a class of tools and utilities, such as operating system kernel, loader, that though needed for the system kernel, loader, that though needed for the applications, are usually present on the remote applications, are usually present on the remote computing node.computing node.

It’s hard to define a clear border between the application It’s hard to define a clear border between the application and the operating system, so one sometimes has to and the operating system, so one sometimes has to decide what to include into the distribution, and what decide what to include into the distribution, and what must be pre-installed by the local system administratormust be pre-installed by the local system administrator

In CMS software these issues are controlled through the In CMS software these issues are controlled through the projects configuration, which specifies the required tools projects configuration, which specifies the required tools and corresponding environment and corresponding environment



Concept Concept ► ► CConditionalsonditionals

Application software must be Application software must be relocatablerelocatable. . This is most important and natural requirement. This is most important and natural requirement. Most of real quality software products are Most of real quality software products are relocatable, and the location is usually controlled relocatable, and the location is usually controlled through the shell environment variable.through the shell environment variable.No hard-coded absolute paths in the program or No hard-coded absolute paths in the program or in the shared libraries (except those referred to in the shared libraries (except those referred to the system area).the system area).All executables are found in the PATH. All executables are found in the PATH. DAR distributions rely on the system DAR distributions rely on the system compatibility compatibility



DAR Implementation DAR Implementation

DAR is implemented in scripting languages, no DAR is implemented in scripting languages, no compilation is required.compilation is required.

Core of DAR code is written in PERL.Core of DAR code is written in PERL.

Interfaces and extensions are written in Python. Interfaces and extensions are written in Python.

DAR code can be simply download from the DAR code can be simply download from the CVS repository or from the web, and can be CVS repository or from the web, and can be used immediately. In the CMS environment:used immediately. In the CMS environment: dar -c <top release directory> <temporary directory> dar -c <top release directory> <temporary directory>

On the remote site:On the remote site: Dar -i <distribution darball> <installation directory> Dar -i <distribution darball> <installation directory>



Implementation Implementation ►►Shared LibrariesShared Libraries

DAR will walk through the directories specified in DAR will walk through the directories specified in the $LD_LIBRARY_PATH environment variable the $LD_LIBRARY_PATH environment variable and package all found libraries into the and package all found libraries into the distribution. It will insure that upon installation distribution. It will insure that upon installation the runtime environment scripts will set proper the runtime environment scripts will set proper $LD_LIBRARY_PATH in correct order.$LD_LIBRARY_PATH in correct order.

DAR does not rely on the output of the DAR does not rely on the output of the ldd <executable> ldd <executable>

command, as this is considered unsafe in case command, as this is considered unsafe in case of dynamically loaded libraries.of dynamically loaded libraries.



Implementation Implementation ►► Executables Executables

By default DAR will walk through the directories By default DAR will walk through the directories in the $PATH environment variable (only the in the $PATH environment variable (only the portion added for this particular application) and portion added for this particular application) and include the contents of directories into DARball.include the contents of directories into DARball.

This behavior can be overwritten by setting the This behavior can be overwritten by setting the $DAR_runtime_PATH environment variable, in $DAR_runtime_PATH environment variable, in which case the associated files and directories which case the associated files and directories will be included into the distribution, and will be will be included into the distribution, and will be added to the $PATH in the DAR runtime added to the $PATH in the DAR runtime environment scripts. environment scripts.



Implementation Implementation ►► Other Variables Other Variables

DAR distinguishes between three types of the DAR distinguishes between three types of the runtime environment variables:runtime environment variables:

1.1. Simple valuesSimple values (flags) (flags)

2.2. Variables associated with Variables associated with some pathsome path to existing file to existing file or directory in the local file systemor directory in the local file system

3.3. Variables associated with several paths in the local Variables associated with several paths in the local file system file system ($PATH-like variables($PATH-like variables, were entries are , were entries are separated by the colon delimiter)separated by the colon delimiter)

All physical files and directories found in All physical files and directories found in specified paths are included preserving the specified paths are included preserving the underlying directory structure. underlying directory structure.



General Practices, TestsGeneral Practices, Tests

All sophisticated work is done by DAR while All sophisticated work is done by DAR while creating the distributioncreating the distribution

The installation procedure is extremely simple. The installation procedure is extremely simple.

Friendly user interface: simple commands, built-Friendly user interface: simple commands, built-in help, backward compatibility.in help, backward compatibility.



TestsTests

1.1. Run same application in the native environmentRun same application in the native environment

2.2. Install DARball and run on the same nodeInstall DARball and run on the same node

3.3. Install and run application on remote host Install and run application on remote host without pre-installed CMS environmentwithout pre-installed CMS environment

Same output in all three cases means success.Same output in all three cases means success.

Second type of tests is optional, and can be usedSecond type of tests is optional, and can be used

to identify any discrepancies in the operating to identify any discrepancies in the operating

system configuration.system configuration.



Using DAR in ProductionUsing DAR in Production

DAR created distributions have been used as a DAR created distributions have been used as a mandatory way to install software for the official mandatory way to install software for the official CMS Monte Carlo production. CMS Monte Carlo production. Using the same set of applications and Using the same set of applications and consistent software distribution mechanisms consistent software distribution mechanisms insured stable performance and trustworthy insured stable performance and trustworthy results. results. The RefDB2DAR interface has been developed The RefDB2DAR interface has been developed to formalize the requests for applications and to formalize the requests for applications and

provide bookkeeping of the available provide bookkeeping of the available distributions.distributions.



CMS production over GRID (fall 2002)CMS production over GRID (fall 2002)The CMS Integration GRID Testbed produced 1.2 million CMS Monte Carlo events from generation with PYTHIA physics generator through simulation with GEANT and digitization with Objectivity based applications.All results shown here were run on Red Hat 6 systems, though some GEANT-only production was also run on newer Red Hat 7 systems.



Next steps Next steps ► ► BookkeepiingBookkeepiing

The RefDB2DAR interface allows to The RefDB2DAR interface allows to download request file from the RefDB. download request file from the RefDB.

Refdbdar utility is then used to Refdbdar utility is then used to parse and validate the RefDB request fileparse and validate the RefDB request file call Packager: CMSIM_packager, call Packager: CMSIM_packager,

CMKIN_packager, or DAR_packager for CMKIN_packager, or DAR_packager for scram managed projects, scram managed projects,

packager builds executables as requested packager builds executables as requested and creates distributionand creates distribution



Next steps Next steps ► ► OptimizationsOptimizations

runtime environment contains some superfluous directories and files. However for detection of files, that could be safely excluded, expert's knowledge of the software application is required.

a number of new expert options allow to filter the contents, but it may take several iterations to figure out what can be removed, and whether it is efficient and safe.



Next steps Next steps ► ► OptimizationsOptimizations

Space optimizations: Avoid duplications (all duplicated files are replaced

by symbolic links)

Introduced expert’s options : Runtime environment contains some superfluous

directories and files. However for detection of files, that could be safely

excluded, expert's knowledge of the software application is required

Time optimizations: Automating tests



Distribution processDistribution process

1.1. Production Coordinator fills web form to create Production Coordinator fills web form to create DARball request. Generated request is stored DARball request. Generated request is stored in the RefDB, notification is sent by e-mail.in the RefDB, notification is sent by e-mail.

2.2. DARball is created then created using refdbdar DARball is created then created using refdbdar and request file, based on software release and request file, based on software release installation at CERN.installation at CERN.

3.3. Application is installed and tested in DAR Application is installed and tested in DAR runtime environment.runtime environment.



Distribution processDistribution process

4.4. DARball is put into SRB for distribution and is DARball is put into SRB for distribution and is ready for the production assignments. ready for the production assignments.

5.5. Production sites get the assignments with the Production sites get the assignments with the indication of the DARball (by name). DARball is indication of the DARball (by name). DARball is then downloaded from the SRB and installed, then downloaded from the SRB and installed, using DAR, on the worker nodes.using DAR, on the worker nodes.

6.6. McRunJob tool creates job based on McRunJob tool creates job based on application and submitts it to the production application and submitts it to the production GRID. GRID.



Using DAR in MOPUsing DAR in MOP

MOP is a system for distributing CMS Monte-MOP is a system for distributing CMS Monte-Carlo production jobs over the GRID.Carlo production jobs over the GRID.

MOP has capability of running any type of MOP has capability of running any type of scripts (jobs) at remote GRID sites, called scripts (jobs) at remote GRID sites, called Worker Sites.Worker Sites.

MOP run jobs as DAGs (Decyclic Acrylic Graph) MOP run jobs as DAGs (Decyclic Acrylic Graph) which could be combined together to create which could be combined together to create complex workflows. complex workflows.



Using DAR in MOPUsing DAR in MOP

In general every DAG contain In general every DAG contain 04 stages.04 stages.

Stage-in: Bring in the Stage-in: Bring in the required input files (from required input files (from several sources) to worker several sources) to worker site.site.

Run: Execute the job itself, Run: Execute the job itself, producing results, logs, producing results, logs, data.data.

Stage-out: Send out Stage-out: Send out produced results/data/logs.produced results/data/logs.

Clean-up: clean the left Clean-up: clean the left over files/directories at over files/directories at worker site.worker site.

stage-in

run

stage-out

clean-up



Using DAR with MOPUsing DAR with MOP

DAR installation at a worker site is achieved byDAR installation at a worker site is achieved by creating a special MOP job creating a special MOP job that first pull DAR tool and Application DAR that first pull DAR tool and Application DAR

distribution in stage-in, distribution in stage-in, runs installation by invoking DAR in run-stage, runs installation by invoking DAR in run-stage, Bring back the results of installation to submission Bring back the results of installation to submission

site in stage-out site in stage-out and then performs a clean up operations at worker and then performs a clean up operations at worker

site.site.



SummarySummary

DAR-based distribution scheme is successfully DAR-based distribution scheme is successfully used in the CMS event production for an used in the CMS event production for an extended period of time. extended period of time.

It allows to keep the pace with the software It allows to keep the pace with the software developments and deliver software applications developments and deliver software applications to the productions sites with ease and in a timely to the productions sites with ease and in a timely fashion.fashion.

Being re-packaged into RPM files, applications Being re-packaged into RPM files, applications can be re-used within different distribution can be re-used within different distribution approaches (e.g. LCFG). approaches (e.g. LCFG).



AcknowledgementsAcknowledgements

Main credit in this work should be addressed to the core Main credit in this work should be addressed to the core CMS software developers, architects and release CMS software developers, architects and release managers for the constant care about software quality.managers for the constant care about software quality.

We would like to thank CMS and USCMS software and We would like to thank CMS and USCMS software and computing managers for their attention paid to this computing managers for their attention paid to this project, CMS Production Team for providing excellent project, CMS Production Team for providing excellent working environment, and all CMS colleagues from many working environment, and all CMS colleagues from many counties and institutions for their useful feedback.counties and institutions for their useful feedback.

My special thanks to Dr. Yujun Wu for presenting this My special thanks to Dr. Yujun Wu for presenting this talk to You, and numerous fruitful discussions.talk to You, and numerous fruitful discussions.

THANK YOUTHANK YOU

4 December 2003 Natalia Ratnikova Natalia Ratnikova 1 Software Packaging with DAR Natalia Ratnikova,...

Documents

Transcript of 4 December 2003 Natalia Ratnikova Natalia Ratnikova 1 Software Packaging with DAR Natalia Ratnikova,...