IceTray Part I: For Users - DESY...22.11.2010 Fabian Kislat - IceTray I: For Users 19 Creating an...
55
IceTray Part I: For Users Fabian Kislat, DESY 22.11.2010
Transcript of IceTray Part I: For Users - DESY...22.11.2010 Fabian Kislat - IceTray I: For Users 19 Creating an...
IceTray Part I:For Users
Fabian Kislat, DESY22.11.2010
22.11.2010 Fabian Kislat - IceTray I: For Users 2
What this talk is:A starting point for using icetray.A brief overview over the features.
What this talk is not:An in depth tutorial to all ofthe IceCube software.
22.11.2010 Fabian Kislat - IceTray I: For Users 3
IceTray Concepts
A simple I3Tray
Extracting data for analysis
Applying cuts
Installing IceRec
22.11.2010 Fabian Kislat - IceTray I: For Users 4
The Tray
● I3Tray: IceTray framework
● I3Tray: Series of modules
● I3Module:● Read data● Generate Monte Carlo● Do physics● Write data● TrashCan
● I3Tray: Carry data from module to module
● I3Frame: Container for data
● Implemented in Python script
I3Tray
I3Module: Source
I3Module
I3Module
I3Module
I3Module: Write
TrashCan
I3Frame: ●Data
I3Frame: ●Data●Data
●Data
I3Frame: ●Data●Data
●Data●Data
I3Frame: ●Data●Data
●Data●Data
●Data
I3Frame: ●Data●Data
●Data●Data
●Data
22.11.2010 Fabian Kislat - IceTray I: For Users 5
The I3Frame
● Data container
● Objects stored under unique name
● Write once read many: Users can● put new data in frame● read data in frame● delete data from frame● not change data in frame
I3Frame
● “I3Calibration”⇒ I3Calibration
● “I3DetectorStatus”⇒ I3DetectorStatus
● “I3Geometry”⇒ I3Geometry
● “LineFitParams”⇒ I3LineFitParams
● “MuonPulseSeriesReco”⇒ I3RecoPulseSeriesMap
● “TopEvent_0”⇒ I3RecoPulseSeriesMap
22.11.2010 Fabian Kislat - IceTray I: For Users 6
Creating an I3Tray
● Example Python script
● Load I3Tray module
● Create an instance of I3Tray object
#!/usr/bin/env python
from I3Tray import *
tray = I3Tray()
22.11.2010 Fabian Kislat - IceTray I: For Users 7
Creating an I3Tray#!/usr/bin/env python
from I3Tray import *
tray = I3Tray()
Make the script executableEnsure using correct python
22.11.2010 Fabian Kislat - IceTray I: For Users 8
Creating an I3Tray
Import all functions/classes from I3Tray module
#!/usr/bin/env python
from I3Tray import *
tray = I3Tray()
22.11.2010 Fabian Kislat - IceTray I: For Users 9
Creating an I3Tray
Create instance ofI3Tray object
#!/usr/bin/env python
from I3Tray import *
tray = I3Tray()
22.11.2010 Fabian Kislat - IceTray I: For Users 10
Creating an I3Tray
● Example Python script
● Load I3Tray module
● Create an instance of I3Tray object
● Add module to read input file
#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )
22.11.2010 Fabian Kislat - IceTray I: For Users 11
Creating an I3Tray#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )
Add a module tothe tray
22.11.2010 Fabian Kislat - IceTray I: For Users 12
I3Reader = type of module
Creating an I3Tray
Add a module tothe tray
#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )
22.11.2010 Fabian Kislat - IceTray I: For Users 13
I3Reader = type of module
Creating an I3Tray
Add a module tothe tray
#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )
Every module on the trayhas a distinct name
22.11.2010 Fabian Kislat - IceTray I: For Users 14
I3Reader = type of module
Creating an I3Tray
Add a module tothe tray
#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )
Every module on the trayhas a distinct name
Module option:Input file name
22.11.2010 Fabian Kislat - IceTray I: For Users 15
Creating an I3Tray#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )
Before you we can actuallyuse I3Reader, we have to
import the dataio module.Module icetray is always needed
22.11.2010 Fabian Kislat - IceTray I: For Users 16
Creating an I3Tray
● Example Python script
● Load I3Tray module
● Create an instance of I3Tray object
● Add module to read input file
● Add a Physics module
#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio, linefit
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )tray.AddModule(“I3LineFit”, “linefit”, InputRecoPulses = “MuonPulses” )
22.11.2010 Fabian Kislat - IceTray I: For Users 17
Creating an I3Tray#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio, linefit
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )tray.AddModule(“I3LineFit”, “linefit”, InputRecoPulses = “MuonPulses” )
Import the required module
22.11.2010 Fabian Kislat - IceTray I: For Users 18
Creating an I3Tray#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio, linefit
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )tray.AddModule(“I3LineFit”, “linefit”, InputRecoPulses = “MuonPulses” )
Add I3LineFit moduleto our I3Tray
22.11.2010 Fabian Kislat - IceTray I: For Users 19
Creating an I3Tray#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio, linefit
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )tray.AddModule(“I3LineFit”, “linefit”, InputRecoPulses = “MuonPulsesReco” )
Load the requiredshared library
Add I3LineFit moduleto our I3Tray
Tell linefit what data itshould operate on
22.11.2010 Fabian Kislat - IceTray I: For Users 20
Creating an I3Tray
● Example Python script
● Load I3Tray module
● Create an instance of I3Tray object
● Add module to read input file
● Add a Physics module
● Write data to disk
#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio, linefit
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )tray.AddModule(“I3LineFit”, “linefit”, InputRecoPulses = “MuonPulsesReco” )tray.AddModule(“I3Writer”, “writer”, FileName = “output.i3.gz” )
22.11.2010 Fabian Kislat - IceTray I: For Users 21
Creating an I3Tray#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio, linefit
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )tray.AddModule(“I3LineFit”, “linefit”, InputRecoPulses = “MuonPulsesReco” )tray.AddModule(“I3Writer”, “writer”, FileName = “output.i3.gz” )
Add I3Writer module toI3Tray and specify output
file name
Module dataio alsocontains I3Writer
22.11.2010 Fabian Kislat - IceTray I: For Users 22
Creating an I3Tray
● Example Python script
● Load I3Tray module
● Create an instance of I3Tray object
● Add module to read input file
● Add a Physics module
● Write data to disk
● Add TrashCan
● Execute!
#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio, linefit
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )tray.AddModule(“I3LineFit”, “linefit”, InputRecoPulses = “MuonPulsesReco” )tray.AddModule(“I3Writer”, “writer”, FileName = “output.i3.gz” )tray.AddModule(“TrashCan”, “trash”)
tray.Execute()tray.Finish()
22.11.2010 Fabian Kislat - IceTray I: For Users 23
Creating an I3Tray#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio, linefit
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )tray.AddModule(“I3LineFit”, “linefit”, InputRecoPulses = “MuonPulsesReco” )tray.AddModule(“I3Writer”, “writer”, FileName = “output.i3.gz” )tray.AddModule(“TrashCan”, “trash”)
tray.Execute()tray.Finish()
Add a TrashCanIt is just a special I3Module
22.11.2010 Fabian Kislat - IceTray I: For Users 24
Creating an I3Tray#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio, linefit
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )tray.AddModule(“I3LineFit”, “linefit”, InputRecoPulses = “MuonPulsesReco” )tray.AddModule(“I3Writer”, “writer”, FileName = “output.i3.gz” )tray.AddModule(“TrashCan”, “trash”)
tray.Execute()tray.Finish()
Run the data processingAlternative form: tray.Execute(100)for testing
22.11.2010 Fabian Kislat - IceTray I: For Users 25
Creating an I3Tray#!/usr/bin/env python
from I3Tray import *
from icecube import icetrayfrom icecube import dataio, linefit
tray = I3Tray()
tray.AddModule(“I3Reader”, “reader”, FileName = “input.i3.gz” )tray.AddModule(“I3LineFit”, “linefit”, InputRecoPulses = “MuonPulsesReco” )tray.AddModule(“I3Writer”, “writer”, FileName = “output.i3.gz” )tray.AddModule(“TrashCan”, “trash”)
tray.Execute()tray.Finish()
Finish: Close files, general cleanup
22.11.2010 Fabian Kislat - IceTray I: For Users 26
Using the script
● Make the script executable$ chmod +x myscript.py
● Run it:$ ~/scratch/icerec/build/env-shell.sh myscript.py
Assuming IceRec is installed in~/scratch/icerec/build
env-shell.sh sets up shellenvironment, like PATH,LD_LIBRARY_PATH, …
Run your script
Here you can passfurther parameters
to your script
22.11.2010 Fabian Kislat - IceTray I: For Users 27
I3Tray Summary I
● I3Tray = IceCube's framework for data processing and simulation
● I3Module = Software to do one specific job, e.g.● Calibrate data, e.g. DOMcalibrator, topeventbuilder● Waveform processing, e.g. FeatureExtractor, NFE, topwaveprocessor● Apply reconstruction, e.g. linefit, credo, toprec● Simulation, e.g. pmt-simulator, DOMsimulator, mmc
● Service = Provides some functionality to be used by modules● For instance: Random numbers, Database access
● Main use: processing
● IceTray is not good for making plots
22.11.2010 Fabian Kislat - IceTray I: For Users 28
dataio-shovel / icetray-inspect
● Inspect file contents with dataio-shovel
● Event viewer glshovel or pythen-eventviewer
● Files must be unzipped
● Listing module parameters: icetray-inspect
Demonstration
22.11.2010 Fabian Kislat - IceTray I: For Users 29
Extracting data
● IceTray not well-suited for analysis
● Extract data to ROOT or hdf5 format for analysis with other tools
● IceTray's tool for the job: tableio● For ROOT files: rootwriter● For hdf5 files: hdfwriter
● You can book everything that comes with a converter
● More info at: http://software.icecube.wisc.edu/offline-software.trunk/projects/tableio/ [Link]
22.11.2010 Fabian Kislat - IceTray I: For Users 30
tableio examplefrom I3Tray import *from icecube import icetray, dataclasses, linefit, tableiofrom icecube.tableio import I3TableWriterfrom icecube.rootwriter import I3ROOTTableServicefrom icecube.hdfwriter import I3HDFTableService
22.11.2010 Fabian Kislat - IceTray I: For Users 31
tableio examplefrom I3Tray import *from icecube import icetray, dataclasses, linefit, tableiofrom icecube.tableio import I3TableWriterfrom icecube.rootwriter import I3ROOTTableServicefrom icecube.hdfwriter import I3HDFTableService
Make classes/functions from icetrayand dataclasses known to python
22.11.2010 Fabian Kislat - IceTray I: For Users 32
tableio examplefrom I3Tray import *from icecube import icetray, dataclasses, linefit, tableiofrom icecube.tableio import I3TableWriterfrom icecube.rootwriter import I3ROOTTableServicefrom icecube.hdfwriter import I3HDFTableService
Import the tableio module
22.11.2010 Fabian Kislat - IceTray I: For Users 33
tableio examplefrom I3Tray import *from icecube import icetray, dataclasses, tableio, linefitfrom icecube.tableio import I3TableWriterfrom icecube.rootwriter import I3ROOTTableServicefrom icecube.hdfwriter import I3HDFTableService
For your convenience: importI3TableWriter to toplevel namespace
22.11.2010 Fabian Kislat - IceTray I: For Users 34
tableio examplefrom I3Tray import *from icecube import icetray, dataclasses, linefit, tableiofrom icecube.tableio import I3TableWriterfrom icecube.rootwriter import I3ROOTTableServicefrom icecube.hdfwriter import I3HDFTableService
Import rootwriter and/or hdfwriter
22.11.2010 Fabian Kislat - IceTray I: For Users 35
tableio examplefrom I3Tray import *from icecube import icetray, dataclasses, linefit, tableiofrom icecube.tableio import I3TableWriterfrom icecube.rootwriter import I3ROOTTableServicefrom icecube.hdfwriter import I3HDFTableService
tray = I3Tray()...
The usual stuff: create I3Tray,add modules, ....
22.11.2010 Fabian Kislat - IceTray I: For Users 36
tableio examplefrom I3Tray import *from icecube import icetray, dataclasses, linefit, tableiofrom icecube.tableio import I3TableWriterfrom icecube.rootwriter import I3ROOTTableServicefrom icecube.hdfwriter import I3HDFTableService
tray = I3Tray()...
root_service = I3ROOTTableService(“output.root”)hdf_service = I3HDFTableService(“output.hd5”)
I3ROOTTableService and I3HDFTableService create output files
22.11.2010 Fabian Kislat - IceTray I: For Users 37
tableio examplefrom I3Tray import *from icecube import icetray, dataclasses, linefit, tableiofrom icecube.tableio import I3TableWriterfrom icecube.rootwriter import I3ROOTTableServicefrom icecube.hdfwriter import I3HDFTableService
tray = I3Tray()...
root_service = I3ROOTTableService(“output.root”)hdf_service = I3HDFTableService(“output.hd5”)
tray.AddModule(I3TableWriter, “writer”, Keys = [ “LineFitParams”, “LineFit” ], TableService = [ root_service, hdf_service ] )
Add I3TableWriter
22.11.2010 Fabian Kislat - IceTray I: For Users 38
tableio examplefrom I3Tray import *from icecube import icetray, dataclasses, linefit, tableiofrom icecube.tableio import I3TableWriterfrom icecube.rootwriter import I3ROOTTableServicefrom icecube.hdfwriter import I3HDFTableService
tray = I3Tray()...
root_service = I3ROOTTableService(“output.root”)hdf_service = I3HDFTableService(“output.hd5”)
tray.AddModule(I3TableWriter, “writer”, Keys = [ “LineFitParams”, “LineFit” ], TableService = [ root_service, hdf_service ] )
No quotes!
22.11.2010 Fabian Kislat - IceTray I: For Users 39
tableio examplefrom I3Tray import *from icecube import icetray, dataclasses, linefit, tableiofrom icecube.tableio import I3TableWriterfrom icecube.rootwriter import I3ROOTTableServicefrom icecube.hdfwriter import I3HDFTableService
tray = I3Tray()...
root_service = I3ROOTTableService(“output.root”)hdf_service = I3HDFTableService(“output.hd5”)
tray.AddModule(I3TableWriter, “writer”, Keys = [ “LineFitParams”, “LineFit” ], TableService = [ root_service, hdf_service ] )
Define output file contents
22.11.2010 Fabian Kislat - IceTray I: For Users 40
tableio examplefrom I3Tray import *from icecube import icetray, dataclasses, linefit, tableiofrom icecube.tableio import I3TableWriterfrom icecube.rootwriter import I3ROOTTableServicefrom icecube.hdfwriter import I3HDFTableService
tray = I3Tray()...
root_service = I3ROOTTableService(“output.root”)hdf_service = I3HDFTableService(“output.hd5”)
tray.AddModule(I3TableWriter, “writer”, Keys = [ “LineFitParams”, “LineFit” ], TableService = [ root_service, hdf_service ] )
Use table servicesdefined above
22.11.2010 Fabian Kislat - IceTray I: For Users 41
tableio examplefrom I3Tray import *from icecube import icetray, dataclasses, linefit, tableiofrom icecube.tableio import I3TableWriterfrom icecube.rootwriter import I3ROOTTableServicefrom icecube.hdfwriter import I3HDFTableService
tray = I3Tray()...
root_service = I3ROOTTableService(“output.root”)hdf_service = I3HDFTableService(“output.hd5”)
tray.AddModule(I3TableWriter, “writer”, Keys = [ “LineFitParams”, “LineFit” ], TableService = [ root_service, hdf_service ] )
tray.AddModule(“TrashCan”, “trash”)tray.Execute()tray.Finish()
22.11.2010 Fabian Kislat - IceTray I: For Users 42
Applying Cuts
● Use python to make an event selection
● Simple function that● Takes an I3rame as argument● Returns
– True to keep the event– False to reject it
22.11.2010 Fabian Kislat - IceTray I: For Users 43
Cut example
● Select only events where IceTop ShowerCombined converged
from icecube import icetray, dataclasses
def selection(frame): if “ShowerCombined” in frame: particle = frame[“ShowerCombined”] return particle.FitStatus == dataclasses.I3Particle.OK return False
We need to use objects in python
22.11.2010 Fabian Kislat - IceTray I: For Users 44
Cut example
● Select only events where IceTop ShowerCombined converged
from icecube import icetray, dataclasses
def selection(frame): if “ShowerCombined” in frame: particle = frame[“ShowerCombined”] return particle.FitStatus == dataclasses.I3Particle.OK return False
Function called selectionthat takes frame as argument
22.11.2010 Fabian Kislat - IceTray I: For Users 45
Cut example
● Select only events where IceTop ShowerCombined converged
from icecube import icetray, dataclasses
def selection(frame): if “ShowerCombined” in frame: particle = frame[“ShowerCombined”] return particle.FitStatus == dataclasses.I3Particle.OK return False
A frame acts like a dictionaryCheck if object named
“ShowerCombined” is in frame
22.11.2010 Fabian Kislat - IceTray I: For Users 46
Cut example
● Select only events where IceTop ShowerCombined converged
from icecube import icetray, dataclasses
def selection(frame): if “ShowerCombined” in frame: particle = frame[“ShowerCombined”] return particle.FitStatus == dataclasses.I3Particle.OK return False
If object exists:Get it from the frame
22.11.2010 Fabian Kislat - IceTray I: For Users 47
Cut example
● Select only events where IceTop ShowerCombined converged
from icecube import icetray, dataclasses
def selection(frame): if “ShowerCombined” in frame: particle = frame[“ShowerCombined”] return particle.FitStatus == dataclasses.I3Particle.OK return False
Check fit status
22.11.2010 Fabian Kislat - IceTray I: For Users 48
Cut example
● Select only events where IceTop ShowerCombined converged
from icecube import icetray, dataclasses
def selection(frame): if “ShowerCombined” in frame: particle = frame[“ShowerCombined”] return particle.FitStatus == dataclasses.I3Particle.OK return False
If object not there: return False
22.11.2010 Fabian Kislat - IceTray I: For Users 49
Cut example
● Select only events where IceTop ShowerCombined converged
from icecube import icetray, dataclasses
def selection(frame): if “ShowerCombined” in frame: particle = frame[“ShowerCombined”] return particle.FitStatus == dataclasses.I3Particle.OK return False
... Create I3Tray; add modules; ...
22.11.2010 Fabian Kislat - IceTray I: For Users 50
Cut example
● Select only events where IceTop ShowerCombined converged
from icecube import icetray, dataclasses
def selection(frame): if “ShowerCombined” in frame: particle = frame[“ShowerCombined”] return particle.FitStatus == dataclasses.I3Particle.OK return False
...
tray.AddModule(selection, “Selection”)
Now you can use the function as a moduleIf it returns False ⇒ frame goes into the trash
If it returns True or None ⇒ processing continues
22.11.2010 Fabian Kislat - IceTray I: For Users 51
Cut example
● Select only events where IceTop ShowerCombined converged
from icecube import icetray, dataclasses
def selection(frame): if “ShowerCombined” in frame: particle = frame[“ShowerCombined”] return particle.FitStatus == dataclasses.I3Particle.OK return False
...
tray.AddModule(selection, “Selection”) # skip frames
or tray.AddModule(“SomeOtherModule”, “conditional_module”, If = selection ) Use the python function to conditionally
execute another module
22.11.2010 Fabian Kislat - IceTray I: For Users 52
An IceTray Glossary
● A “project” is a collection of one or more modules for a specific task● linefit contains module I3Linefit● toprec contains modules I3TopRecoCore, I3TopLateralfit, …
● IceRec and Simulation are “meta-projects”● Collection of several modules● Both contain meta-project offline-software as a subset
● All software hosted in IceCube subversion repository:http://www.code.icecube.wisc.edu/svn
● 3rd party software maintained in I3Ports distribution
22.11.2010 Fabian Kislat - IceTray I: For Users 53
Installing IceRec
● First, you need I3Ports:http://software.icecube.wisc.edu/offline-software.trunk/projects/cmake/installing_ports.html [Link]
● At DESY a central installation of I3Ports is maintained by Robert
● All you need:$ . /afs/ifh.de/group/amanda/icecube/ports-v3_test/ports_setup.zsh
● Create a source directory (in your scratch, I would suggest):$ cd scratch && mkdir IceRec$ cd IceRec
● Check out the source code (into directory src):$ svn co \ http://code.icecube.wisc.edu/svn/meta-projects/icerec/trunk src
22.11.2010 Fabian Kislat - IceTray I: For Users 54
Installing IceRec
● Create a build directory:$ mkdir build-release$ cd build-release
● Configure build by running cmake (creates an optimized build):$ $I3_PORTS/bin/cmake ../src -DCMAKE_BUILD_TYPE:String=Release
● Compile:$ nice make -j9
● Always use nice for long-running jobs
● -jX specifies the number of parallel builds
● Good choice: X = #cores + 1● Get a coffee.
● Enjoy!
22.11.2010 Fabian Kislat - IceTray I: For Users 55
Useful miscellanea
To wrap up, here is a list of useful things to know
● For debugging:tray.AddModule(“Dump”, “dump”)Outputs current frame contents
● IceRec documentation:http://www-zeuthen.desy.de/~kislat/www/icerec/docs/trunk/
Contains complete offline-software documentation + IceRec modules
● Mailing lists:● [email protected]
● Join us on IRC: #icecube and/or #i3berlin(see also http://wiki.icecube.wisc.edu/index.php/Join_us_on_IRC)
