LHCb Software week 24-26 November, 1999 M.Frank LHCb/CERN N-Tuples within Gaudi ã Definition ã...

LHCb Software week 24-26 November, 1999

M.Frank LHCb/CERN

N-Tuples within Gaudi

Definition Usage

Run through simplified example

The persistent back-end

M.Frank LHCb/CERNGaudi N-Tuples

PersistencySvc

PObjPObj

PObj

ConverterConverter

Converter TObjTObj

Obj2

TObj

TObjContainerTObjContainer

ObjContainer

EventDataSvc

Transient Event Store

TObjObj1

N-tuple Service

EventSelector

MessageSvc

JobOptionsSvc

AnotherPercySvc

PObjPObj

PObj

What are we talking about?

AppManager

TObj1Obj1

PObjectPObjectPDetElem

DetDataSrv

TDetElem1TDetElem1

TDetElem1

T Detector Store

T Histogram Store

HistogramSvc

Hist1Hist1

Hist1

Algorithm1Algorithm1

Algorithm1

DetPerstySvc

ConverterConverter

Converter

Alg PropertiesAlgFactory

uses

creates

navigability

HistPerstySvc PHistPHist

Converter

PHistPNTup

Converter

ConversionSvc

T N-Tuple Store

NTupleSvc

NTupNTup

NTup1


What was an N-Tuple in CERNLIB

Collection of identifiable n-dim. primitive items LOGICAL*4, INTEGER*4, REAL*4, REAL*8

See HBOOKRow wise N-Tuples

REAL*4 array(SOME_DIMENSION)Items are single numbers identified by tags

Column wise N-TuplesStructure mapped to a common blockItems (number, array, …) are identified by tagsAllow for variable size arrays


What is an N-Tuple in Gaudi

Collection of N-tuple ItemsItems represent primitives: bool, long, float, double

N-tuples reside in a data storeCan be accessed like any other DataObject

Data management (e.g. writing records) is done by the N-tuple service

Things did not change dramatically


What’s the job: Wishes

Stick to architectureWe want to independent from the “back-end”

storage mechanism (HBOOK, ROOT, OBJY...)• Cernlib will die

Encapsulate implementation: Interfaces or ABC

Easy to use• Items must behave like ordinary numbers• Must support basic arithmetic operations and type

conversions

N-tuples should not go all into one single file• possibly, but not always• Connect several streams to the data store


The N-Tuple Data Store

Top Entry point /NTUPLESLogical file names

• User defined• First useable

directory/MC/REC/USER

Directory structureN-tuples

• 1,2,3,4


The Example

Simplified version of GaudiExample/Ntuples

Explain how to write and read a column wise N-tuple

Differences to row wise N-tuples are minor Only record structure differs

N-tuples should be easy to use Tell me if you think stuff is overcomplicated

I do not use them daily


Easy to Use… (Hopefully)

Define data itemse.g. as members of filling Algorithm

Items are based on primitives• bool, long, float, double

// Items for n-tuple // _________________ // // 0 dimensional item (=number) NTuple::Item<long> m_ntrk; // 1 dimensional items (Arrays) NTuple::Array<float> m_px, m_py, m_pz; // 2 dimensional items (Matrices) NTuple::Matrix<long> m_hits;


NTuplePtr nt(ntupleService(),"/NTUPLES/MC/1"); if ( !nt ) { nt = ntupleService()->book("/NTUPLES/MC/1”,

CLID_ColumnWiseTuple, "Hello World");

}

NTuplePtr nt(ntupleService(),"/NTUPLES/MC/1"); if ( !nt ) { nt = ntupleService()->book("/NTUPLES/MC/1”,

CLID_ColumnWiseTuple, "Hello World"); if ( nt ) { status = nt->addItem ("Ntrack", m_ntrk, 0, 5000 ); status = nt->addItem ("px", m_ntrk, m_px); // m_hits[0:m_ntrk][0:5]; numbers within [0,8] status = nt->addItem ("hit", m_ntrk, m_hits, 5, 0, 8 ); }}


Book and add the items you later want to fill

•Book the N-tuple

•Add items


for(m_ntrk=0; m_ntrk < numTracks; m_ntrk++) { if ( m_ntrk >= m_ntrk->range().distance() ) break;

}

for(m_ntrk=0; m_ntrk < numTracks; m_ntrk++) { if ( m_ntrk >= m_ntrk->range().distance() ) break; m_px[m_ntrk] = track[m_ntrk]->px(); m_hits[m_ntrk][0] = track[m_ntrk]->adcValue(0); ... m_hits[m_ntrk][4] = track[m_ntrk]->adcValue(4);}

for(m_ntrk=0; m_ntrk < numTracks; m_ntrk++) { if ( m_ntrk >= m_ntrk->range().distance() ) break; m_px[m_ntrk] = track[m_ntrk]->px(); m_hits[m_ntrk][0] = track[m_ntrk]->adcValue(0); ... m_hits[m_ntrk][4] = track[m_ntrk]->adcValue(4);}// Commit record (here of a column wise N-tuple)status = ntupleService()->writeRecord("/NTUPLES/MC/1");


Fill all items and commit recordSimple calculations can be done directly using the

items

•Fill items

•Commit record

•Do not overrun dimensions

Reset values to defaults: •Is this good?



At the end of the jobAll N-tuples will automatically be closed

The directory structure of the data store is reflected by RZ directoriesAccess N-tuple for reading

in the same way as for writing

Don’t forget to “cd //<lun>/<dir>” in PAW


NTuple::Item<long> ntrk; NTuple::Array<float> px, py, pz; NTupleDirPtr nt(ntupleService(), "/NTUPLES/MC/1"); if ( nt ) {

}

NTuple::Item<long> ntrk; NTuple::Array<float> px, py, pz; NTupleDirPtr nt(ntupleService(), "/NTUPLES/MC/1"); if ( nt ) { status = nt1->item ("Ntrack", ntrk ); status = nt1->item ("px", px); ...

}

NTuple::Item<long> ntrk; NTuple::Array<float> px, py, pz; NTupleDirPtr nt(ntupleService(), "/NTUPLES/MC/1"); if ( nt ) { status = nt1->item ("Ntrack", ntrk ); status = nt1->item ("px", px); ... do { float sumPx = 0; if(ntupleService()->readRecord(nt.ptr()).isSuccess()){ for ( long j = 0; j < ntrk; j++ ) sumPx += px[j]; } } while ( status.isSuccess() ); }


Reading back (…or with PAW)

•Will open the corresponding logical file

•Search for the directory

•Load the N-tuple

•Assign the entries to the N tuple

•Loop over records



Configuration (using job options file)Reading N-tuples

NTupleSvc.Input = { ”MC#tuple1.hbook", ”USER#tuple2.hbook" };// Persistency type of the N-tuple service: 6=HBOOKNTupleSvc.Type = 6;

NTupleSvc.Output = { ”MC#tuple1.hbook", ”USER#tuple2.hbook" };// Persistency type of the N-tuple service: 6=HBOOKNTupleSvc.Type = 6;

Writing N-tuples


Back-End: HBOOK Limitations

Limited type informationCannot define all “C” data types

Therefor only: bool, long, unsigned long, float and double

NOT: char, short, int (+unsigned)

Does not affect disk space: HBOOK compresses internally

N-tuple name is an “integer”“integer” translates later to HBOOK ID

Must be constant independent of booking time for kumacs

Do not mix up N-tuple IDs and Histogram IdsThere is only one PAWC common block!


Back-End: HBOOK Limitations

Only one index variable per itemRZ directory names are CHARACTER*8

… and upper case ONLY

double/REAL*8 dataPAW has trouble: better don’t useREAL*8 must be quadword aligned (64 bits) Padding must be done by userNot more than 1000 REAL*8 entries per N tuple

Row wise N-tuplesType information is stored as first “event”Start reading at the second...


Other Back-Ends

The Present: HBOOKPAW is the most widely used viewer

N-tuples don’t help if they can’t be analyzed interactively

The Future (?): ROOT I/OWill give ROOT a try as back-end

Implementation with trees/branches is straight forward

Would solve probably all the limitations of HBOOK


Conclusions

It is possible to define dynamic structures

(N-tuples) with Gaudi

If you think the usage is too complicated,

please say so!

Otherwise it will never change

Depending on the back-end,

limitations invade transient side

LHCb Software week 24-26 November, 1999 M.Frank LHCb/CERN N-Tuples within Gaudi ã Definition ã...

Documents

Transcript of LHCb Software week 24-26 November, 1999 M.Frank LHCb/CERN N-Tuples within Gaudi ã Definition ã...