High level trigger online calibration framework in ALICE

Journal of Physics Conference Series

OPEN ACCESS

High level trigger online calibration framework inALICETo cite this article S R Bablok et al 2008 J Phys Conf Ser 119 022007

View the article online for updates and enhancements

You may also likeCommissioning of the ATLAS High LevelTrigger with single beam and cosmic raysA Di Mattia and the Atlas Collaboration

-

High level trigger configuration andhandling of trigger tables in the CMS filterfarmG Bauer U Behrens V Boyer et al

-

Operational experience with the ALICEHigh Level TriggerArtur Szostak

-

Recent citationsResults from the first p+p runs of theALICE High Level Trigger at LHCKalliopi Kanaki (for the ALICE HLTcollaboration)

-

This content was downloaded from IP address 947417928 on 16012022 at 1514

High Level Trigger Online Calibration framework in

ALICE

Sebastian Robert Bablok1 Oslashystein Djuvsland1 Kalliopi Kanaki1Joakim Nystrand1 Matthias Richter1 Dieter Rohrich1 KyrreSkjerdal1 Kjetil Ullaland1 Gaute Oslashvrebekk1 Dag Larsen1 JohanAlme1 Torsten Alt2 Volker Lindenstruth2 Timm M Steinbeck2Jochen Thader2 Udo Kebschull2 Stefan Bottger2 SebastianKalcher2 Camilo Lara2 Ralf Panse2 Harald Appelshauser3 MateuszPloskon3 Havard Helstrup4 Kristin F Hetland4 Oslashystein Haaland4Ketil Roed4 Torstein Thingnaeligs4 Kenneth Aamodt5 Per ThomasHille5 Gunnar Lovhoiden5 Bernhard Skaali5 Trine Tveter5 IndranilDas6 Sukalyan Chattopadhyay6 Bruce Becker7 Corrado Cicalo7Davide Marras7 Sabyasachi Siddhanta7 Jean Cleymans8 ArturSzostak87 Roger Fearick8 Gareth de Vaux8 Zeblon Vilakazi81 Department of Physics and Technology University of Bergen Norway2 Kirchhoff Institute of Physics Ruprecht-Karls-University Heidelberg Germany3 Institute for Nuclear Physics University of Frankfurt Germany4 Faculty of Engineering Bergen University College Norway5 Department of Physics University of Oslo Norway6 Saha Institute of Nuclear Physics Kolkata India7 INFN Sezione di Cagliari Cittadella Universitaria Cagliari Italy8 UCT-CERN Department of Physics University of Cape Town South Africa

E-mail SebastianBablokuibno

Abstract The ALICE High Level Trigger (HLT ) is designed to perform event analysis ofheavy ion and proton-proton collisions as well as calibration calculations online A large PCfarm currently under installation enables analysis algorithms to process these computationallyintensive tasks The HLT receives event data from all major detectors in ALICE Interfaces tothe various other systems provide the analysis software with required additional informationProcessed results are sent back to the corresponding systems To allow online performancemonitoring of the detectors an interface for visualizing these results has been developed

1 Introduction of the ALICE High Level TriggerALICE is designed to study heavy ion (Pb-Pb) and proton-proton (pp) collisions at an eventrate of up to 1kHz In the Time Projection Chamber (TPC ) the main tracking detector inALICE with the largest data volume the size of a single event is around 75 MByte After ahierarchical selection of Level 0 1 and 2 triggers and combination with the data of the otherrelevant detectors this sums up to a data rate of 25 GBytes To match this amount with theData Acquisition (DAQ) archiving rate of about 125 GBytes the HLT performs online event

International Conference on Computing in High Energy and Nuclear Physics (CHEPrsquo07) IOP PublishingJournal of Physics Conference Series 119 (2008) 022007 doi1010881742-65961192022007

ccopy 2008 IOP Publishing Ltd 1

Figure 1 ALICE systems structure and event data flow

analysis and data reduction For this purpose the HLT receives raw event data of the Front-End-Electronics (FEE ) which are sent as direct copies of the event data by the DAQ-ReadOutReceiver Cards (D-RORC ) during the run In return the HLT provides a Level 3 trigger (eventselection) and efficient event data compression (eg entropy coding) Additionally the HLTallows for selection of Regions-of-Interest (RoI) within an event performance monitoring of theALICE detectors and calculation of new calibration data online As shown in figure 1 the wholesetup is steered and synchronized via the Experiment Control system (ECS ) [1] [2]

To cope with the large processing requirements involved in these tasks the HLT consists of alarge computing farm with several hundred off-the-shelf PCs These computers contain a dualboard equipped with AMD dual core Opteron 2 GHz CPUs 8 GByte of RAM two GigabitEthernet connections and an Infiniband backbone for high throughput communications Anupgrade to quad core CPUs is foreseen

HLT-ReadOut Receiver Cards (H-RORC ) inside dedicated Front-End-Processor (FEP)nodes accept the raw event data and perform a first reconstruction step Detector Algorithms(DA) on additional cluster nodes take over and accomplish the above mentioned tasks Detector-Data-Links (DDL) which transfer data over optical fibers cover the transportation of resultsback to DAQ There the results are stored together with the event data The layout of thecluster nodes presented in figure 2 is matching the structure of the ALICE detectors and thedifferent event analysis steps involved

Dedicated infrastructure nodes are reserved for services and cluster maintenance (e g an8 TByte AFS (Andrew File System) file server and two gateway machines) Portal nodes takecare of the exchange with the other ALICE systems in the online and offline world These portalnodes and their specialized software are the main focus of this article

The latest server version of Ubuntu Linux currently 66 LTS (Long Term Support) servesas operating system inside the cluster An interweaved system of TaskManagers organizes thecluster and steers the tasks on each node internally A dynamic data transport frameworkdesigned after the publishsubscriber principle takes care of the data flow [3] [4] [5] DetectorAlgorithms (DA) of the AliRoot package the analysis framework of ALICE analyze theincoming raw event data and calculate new calibration settings [6] The analysis software worksindependent from the transportation framework which allows the DAs to run in Offline as well


2

Figure 2 HLT architecture and dataflow inside the HLT cluster The clusterorganization matches the structure of theALICE detectors (TPC Transition Radia-tion Detector (TRD) Inner Tracking Sys-tem (ITS) ) and their analysis stepsfrom cluster finding to trigger decisionsand data compression

without any changes This enables result comparison later on [7]The cluster itself is monitored by the SysMES framework (System Management for Networked

Embedded Systems and Clusters) and Lemon (LHC Era Monitoring) [8] Fail safety and theavoidance of single points of failure have been major issues in the design of the cluster

2 HLT Interfaces21 OverviewThe HLT has redundant interfaces to the various other systems in ALICE These include theALICE online systems like the ECS FEE DAQ and the Detector Control System (DCS ) aswell as the ALICE Offline system and the Alice Event Monitoring (AliEve) framework Thelatter one will allow for monitoring ALICE online in the ALICE Control Room (ACR)

For receiving raw event data 365 DDLs from the different detectors in ALICE are connectedto the H-RORCs in the FEPs The data are analyzed inside the HLT cluster These tasksare in detail provision of Trigger decisions selection of RoIs (only the data of the relevantparts are streamed out) lossless data compression (like entropy coding vector quantization inthe TPC data model) [9] [10] These data are sent back to the DAQ-LDCs (DAQ Local DataConcentrator) for permanent storage via 12 DDLs

The interfaces to ECS DCS Offline and AliEve are described in the following subsectionsIn case of a failure of one portal the backup node takes over the corresponding task In the DCSand Offline case the tasks for communication in the two exchange directions (receiving data andsending data) are separated in different applications with own names DCS data are fetched viathe so called Pendolino while HLT data are sent to the DCS over the Front-End-Device (FED)API [11] Offline can fetch data from HLT using the Offline Shuttle mechanism and data fromOffline are retrieved over the HLT Taxi A sketch of these interfaces is presented in figure 3

22 ECS interfaceThe HLT like all other ALICE systems is controlled by the ECS An ECS-proxy consisting of afinite state machine contacts the ECS and informs about its current state Transition commandsissued by ECS trigger state changes and provide the initial settings of the upcoming run Thisinformation includes the upcoming run number the experiment type (Pb-Pb or p-p) operatingmode trigger classes DDL lists etc The ECS-proxy accepts the current state from the MasterTaskManagers which control the HLT cluster internally All state transition commands issuedby ECS are referred to the Master TaskManagers as well [12]

The proxy is implemented in SMI++ (State Management Interface) which communicateswith the ALICE ECS system using DIM (Distributed Information Management) a


3

Figure 3 Overview of the HLT interfaces (FEP = Front-End-Processor DDL = Detector DataLink HOMER = HLT Online Monitoring Environment including ROOT)

communication framework developed at CERN [13] The ECS is connected to all other ALICEsystems This allows for synchronizing the HLT with the other parts of ALICE

23 DCS interface231 Pendolino The DCS controls and configures the FEE of all detectors in ALICE Inaddition the current detector status is monitored their temperature voltage and current valuesare measured These run conditions are received in the PVSS (Process Visualization and SteeringSystem) panels of the corresponding detectors and then stored as datapoints to the DCS ArchiveDB during the run

The DAs running in the HLT cluster require a well defined subset of these values to calculatecalibration settings and observables like the TPC drift velocity Therefore a special HLTapplication the Pendolino contacts the DCS during the run and fetches the desired valuesSince these values are constantly measured and can vary during the run the Pendolino requeststhese values frequently from an Amanda server (Alice MANager for Dcs Archives) which sits ontop of the DCS Archive DB [14] It is foreseen to have three different Pendolinos running eachwith a different frequency and each requesting a different subset of datapoints These datapointsare received as timestamp value pairs To allow the DAs to read the data regardless of whetherrunning online or offline the pairs have to be preprocessed and enveloped into ROOT objectsEach detector providing DAs to the HLT has to implement its own preprocessing routine Thisprocedure is adapted from the Offline Shuttle mechanism which is used to store DCS data into


4

Figure 4 Deployment of the Pendolinofor fetching DCS values from the DCSArchive DB and providing them as ROOTfiles to the DAs in the HLT cluster

the Offline Condition DataBase (OCDB) [15]Due to the fact that it can take up to two minutes until DCS data are shipped from the

PVSS panels to the DCS Archive DB the preprocessing routine for HLT has to encode someprediction calculation for the retrieved values in the produced ROOT objects This might berequired for certain values in order to cope with online processing The prediction encoding givesthe routine its name PredictionProcessor The implementation of the prediction calculation isup to the detectors requiring the data The produced ROOT file is stored in a file cataloguecalled HLT Condition DataBase (HCDB) The file catalogue is distributed to the cluster nodesrunning the DAs and updated each time new data are available Afterwards a notification aboutnew content in the HCDB is percolated through the analysis chain The Pendolino procedureis visualized in figure 4

232 FED-portal To return data like the TPC drift velocity to the DCS system the HLT usesthe Front-End-Device (FED) API which is common among all detectors integrated in the DCSTherefore DCS related data inside the HLT cluster are collected by the FED-portal during therun A DIM server implementing the FED API sends these data from the FED-portal to thecorresponding PVSS panels on the DCS side From there it is included automatically in theDCS Archive DB

24 Offline interface241 Taxi Assumed or in former runs calculated calibration and condition settings are storedas ROOT files in the OCDB [15] The DAs require them in order to analyze events andcalculate new calibration objects A special application called Taxi requests the OCDB forlatest available calibrations settings in regular time intervals and synchronizes them with thelocal copy of the OCDB the HCDB To reduce traffic the Taxi first checks if the data arealready available in the HCDB before it is fetched from the OCDB

The whole procedure runs independently and asynchronously to any run At the start of eachrun the current version of the HCDB is fixed to avoid updates of calibrations settings duringthe run Then the HCDB is distributed to all cluster nodes running DAs

Access to the HCDB is granted through the AliCDB (AliRoot Conditions Database) Accessclasses which are also used in Offline to request the OCDB This guarantees transparent accessfor the DAs independent from running online or offline The AliCDB Access classes returnautomatically the latest version of calibration settings valid for a given run number


5

Figure 5 Deployment of the Shuttleportal and the Offline Shuttle mechanismfor retrieving new calculated calibrationobjects from the HLT cluster

242 Shuttle portal After each run the Shuttle portal collects all newly calculated calibrationobjects from the DAs Transportation of the data are realized via dedicated components of thePublisherSubscriber framework The calibration objects are stored in a File EXchange Server(FXS) while additional meta data for each file (like run number detector file ID file sizechecksum and timestamps) are stored in a MySQL DB When all new objects are saved theShuttle portal notifies the ECS-proxy that the collection process has finished Now the ECScan trigger the start of the Offline Shuttle The Shuttle requests the meta data of the latest runfor the new entries in the FXS from the MySQL DB Then it fetches the according files fromthe Shuttle portal non-interactively using an shh-key All new files are preprocessed by detectorspecific ShuttlePreprocessors and enveloped in ROOT files if not already done inside the HLTcluster [15] Afterwards the new entries are stored in the OCDB where the Taxi can fetch themfor the next run The detour over the OCDB has been chosen to guarantee coherent versioncontrol of the calibration objects The whole mechanism is sketched in fig 5

25 AliEve interfaceSince the HLT performs the task of event analysis and calculation of new calibration data onlineobservation of the results is also possible online Therefore the HLT provides the HOMER (HLTOnline Monitoring Environment including ROOT ) interface which offers a connection to theAlice Event monitoring framework (AliEve) AliEve is part of the AliRoot package and includes3D visualization as well as displaying of ROOT structures and histograms [16]

HOMER can fetch produced results at any step of the HLT analysis chain and transport themto any AliEve application inside the CERN General Purpose Network (GPN) This enables theoperators to display results directly in the ACR

3 Time line sequence and synchronizationEach of the presented interfaces have dedicated places in the usage sequence As shown in figure6 this sequence is divided in five different time periods

bull Independent from a run (asynchronous to the runs)Before a first run (and repeated in regular time intervals) the Taxi requests latest calibration


6

Figure 6 Sequence diagram displaying the interplay of the different interfaces and portalsparticipating in the calibration framework of HLT SoR (Start-of-Run) and EoR (End-of-Run)are special events triggered by ECS to indicate the start and end of a run

settings from the OCDB and caches them locally in the HCDB Actually this task isaccomplished completely asynchronous to any run and can be performed also during a run

bull Initialization period before a runThe ECS informs the HLT about an upcoming run with a first INITIALIZE commandIn addition several run settings (like run number beam type trigger classes etc) aretransmitted During the following configuration steps the HLT freezes the current versionof the HCDB and distributes it to the cluster nodes running the DAs In case the Taxifetches new data from the OCDB during a run the new settings are only stored to theHCDB version located on the Taxi portal node but not updated on the DA nodes Thisguarantees that the DAs use a coherent version during the complete run The completionof the initialization procedure is signaled back to the ECS

bull During a runEvery run starts with a special event triggered by ECS i e the Start-of-Run (SoR) Afterthe SoR event raw event data are received from the FEE on the FEP nodes The dataare processed and analyzed over several steps New calibration settings are calculated Foradditional input the Pendolino fetches current environment and condition settings fromthe DCS Archive DB (like temperature voltages etc) After preprocessing and envelopingthem they are available for the DAs via the HCDB Analyzed events and trigger decisionsare streamed out to DAQ for permanent storage Freshly calculated DCS relevant dataare sent through the FED-portal for monitoring and storage in DCS Online visualizationof events and calibration data is enabled via the HOMER interface and allows to monitorthe performance of the detectors in the ACR This is continuously repeated during the runand a notification about new DCS data in the HCDB is percolated through the analysischain after each update

bull End of a runAt the end of a run ECS issues again a special event called End-of-Run (EoR) The eventis percolated through the analysis chain and notifies each component to terminate Thisphase is called completing because it can take some time until all events are worked offand until the HLT is ready for the next run During this time the Shuttle portal collectsall freshly produced calibration objects fills them in the FXS and stores additional metadata in the MySQL DB As soon as this is finished the ECS-proxy signals to ECS that theOffline Shuttle can start collecting HLT data


7

bull After the endFinally the Offline Shuttle can contact the MySQL DB and the FXS on the correspondingHLT portal node and fetch the new data for the OCDB The HLT cluster can already beused for the next run since the fetching does not require actions from the HLT side

4 Status and PerformanceThese interfaces are in different stage of development Most of them have been implementedand are in the test phase which leads to an ongoing optimization and fine tuning of the differentinterface

The ECS-proxy has been implemented over a year ago and its functionality has been testedwidely in tests with ECS and during the TPC commissioning in June 2007 The Shuttle portaland the Taxi are written and now deployed for performance tests and enhancements Firstmeasurements indicate that they will do their job according to the requirements The Pendolinois implemented without the PredictionProcessor and executing currently performance tests aswell At the moment the Pendolino takes 9 seconds to fetch 250 different datapoints using all 250datapoint names in one request A soon to come upgrade in the Amanda server which avoids adetour in the request over PVSS will bring further speed enhancements The PredictionProcessorinterface is in the final discussion and a first prototype using the TPC PredictionProcessor isabout to be implemented soon The FED-API of the FED-portal is implemented and waiting tobe tested in the PubSub (PublishSubscriber) framework Inclusion in the corresponding PVSSpanels is pending The HOMER has been implemented a while ago and widely tested Last timein the setup of the TPC commissioning The HLT has been able to monitor the TPC onlineduring the commissioning in the ACR the results are very promising A combined test with allinterfaces is pending but scheduled for the full dress rehearsal in beginning of November 2007

5 SummaryThe ALICE HLT consists of a large computing farm with approx 1000 computing units Fastconnections guarantee high performance throughput of data The layout of the cluster matchesthe structure of the ALICE detectors and their analysis steps Interfaces to other parts of ALICEallow for data exchange with online and offline systems Current run conditions are read fromDCS calibration settings fetched from Offline Connections in the vice versa direction allowfor feeding back new data An interface to AliEve allows to visualize processed events onlineExternal cluster control and synchronization is achieved via the ECS-proxy

The framework presented in this article enables the HLT for detector performancemeasurements and physics monitoring as well as calibration calculation online The HLT willbe able to provide all required data for the analysis software performing first physics in ALICE2008

AcknowledgmentsThe development of these HLT interfaces have been accompanied by very good and fruitfulcooperations with the collaborations of the connected systems in ALICE

The ALICE HLT project has been supported by the Norwegian Research Council (NFR)

References[1] ALICE Collaboration ALICE Technical Proposal for A Large Ion Collider Experiment at the CERN LHC

CERNLHCC 1995-71 (1998)[2] ALICE Collaboration ALICE Technical Design Report of the Trigger Data Acquisition High-Level Trigger

and Control System ALICE-TDR-10 CERN-LHCC-2003-062 (2003) pp 245-356[3] Steinbeck T M et al New experiences with the ALICE High Level Trigger Data Transport

Framework in Proc Computing in High Energy Physics Conf 2004 (CHEP04) Interlaken Switzerlandhttpchep2004webcernchchep2004


8

[4] Steinbeck T M et al 2002 An object-oriented network-transparent data transportation framework IEEETrans Nucl Sci 49 (2002) pp 455-459

[5] Steinbeck T M et al 2002 A Framework for Building Distributed Data Flow Chains in Clusters Proc 6thInternational Conference PARA 2002 Espoo Finland June 2002 Lecture Notes in Computer ScienceLNCS 2367 pp 254-464 Springer-Verlag Heidelberg

[6] ALICE Off-line project httpaliceinfocernchOfflineAliRootManualhtml[7] Richter M et al High Level Trigger applications for the ALICE experiment submitted to IEEE Trans Nucl

Sci[8] Lara C The SysMes architecture System management for networked embedded sys-

tems and clusters Date 2007 PhD Forum Nice France (2007) httpwikikipuni-heidelbergdetiSysMESimagesaa5DatePhDForumPosterpdf

[9] Rohrich D and Vestboslash A Efficient TPC data compression by track and cluster modeling Nucl InstrumMeth A566 (2006) pp 668-674

[10] Lindenstruth V et al Real time TPC analysis with the ALICE High Level Trigger Nucl Instrum MethA534 (2004) pp 47-52

[11] httpalicedcswebcernchalicedcsDocumentsFedServerAPIpdf[12] Bablok S et al ALICE HLT interfaces and data organisation Proc Computing in High Energy and Nuclear

Physics Conf 2006 (CHEP 2006) Mumbai India ed Banerjee S vol 1 Macmillian India Ltd (2007) pp96-99

[13] Gaspar C et al An architecture and a framework for the design and implementation of large control systemProc ICALEPS 1999 Trieste Italy

[14] ALICE DCS Amanda project httpalice-project-dcs-amandaserverwebcernchalice-project-dcs-amandaserver

[15] Colla A and Grosse-Oetringhaus J F Alice internal note describing the Offline Shuttle mechanism (about tobe published)

[16] Tadel M and Mrak-Tadel A AliEVE ALICE Event Visualization Environment Proc Computing in HighEnergy and Nuclear Physics Conf 2006 (CHEP 2006) Mumbai India ed Banerjee S vol 1 MacmillianIndia Ltd (2007) pp 398-401


9

High Level Trigger Online Calibration framework in

ALICE

Sebastian Robert Bablok1 Oslashystein Djuvsland1 Kalliopi Kanaki1Joakim Nystrand1 Matthias Richter1 Dieter Rohrich1 KyrreSkjerdal1 Kjetil Ullaland1 Gaute Oslashvrebekk1 Dag Larsen1 JohanAlme1 Torsten Alt2 Volker Lindenstruth2 Timm M Steinbeck2Jochen Thader2 Udo Kebschull2 Stefan Bottger2 SebastianKalcher2 Camilo Lara2 Ralf Panse2 Harald Appelshauser3 MateuszPloskon3 Havard Helstrup4 Kristin F Hetland4 Oslashystein Haaland4Ketil Roed4 Torstein Thingnaeligs4 Kenneth Aamodt5 Per ThomasHille5 Gunnar Lovhoiden5 Bernhard Skaali5 Trine Tveter5 IndranilDas6 Sukalyan Chattopadhyay6 Bruce Becker7 Corrado Cicalo7Davide Marras7 Sabyasachi Siddhanta7 Jean Cleymans8 ArturSzostak87 Roger Fearick8 Gareth de Vaux8 Zeblon Vilakazi81 Department of Physics and Technology University of Bergen Norway2 Kirchhoff Institute of Physics Ruprecht-Karls-University Heidelberg Germany3 Institute for Nuclear Physics University of Frankfurt Germany4 Faculty of Engineering Bergen University College Norway5 Department of Physics University of Oslo Norway6 Saha Institute of Nuclear Physics Kolkata India7 INFN Sezione di Cagliari Cittadella Universitaria Cagliari Italy8 UCT-CERN Department of Physics University of Cape Town South Africa

E-mail SebastianBablokuibno

Abstract The ALICE High Level Trigger (HLT ) is designed to perform event analysis ofheavy ion and proton-proton collisions as well as calibration calculations online A large PCfarm currently under installation enables analysis algorithms to process these computationallyintensive tasks The HLT receives event data from all major detectors in ALICE Interfaces tothe various other systems provide the analysis software with required additional informationProcessed results are sent back to the corresponding systems To allow online performancemonitoring of the detectors an interface for visualizing these results has been developed

1 Introduction of the ALICE High Level TriggerALICE is designed to study heavy ion (Pb-Pb) and proton-proton (pp) collisions at an eventrate of up to 1kHz In the Time Projection Chamber (TPC ) the main tracking detector inALICE with the largest data volume the size of a single event is around 75 MByte After ahierarchical selection of Level 0 1 and 2 triggers and combination with the data of the otherrelevant detectors this sums up to a data rate of 25 GBytes To match this amount with theData Acquisition (DAQ) archiving rate of about 125 GBytes the HLT performs online event


ccopy 2008 IOP Publishing Ltd 1








2










3






4









5








6







7













8















9








2










3






4









5








6







7













8















9










3






4









5








6







7













8















9






4









5








6







7













8















9









5








6







7













8















9








6







7













8















9







7













8















9













8















9















9

High level trigger online calibration framework in ALICE

Documents

Transcript of High level trigger online calibration framework in ALICE