Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow...
Transcript of Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow...
ESO Pipeline Workshop, Oct 24 2014
Reflex workflow developement
Enrique GarciaESO Pipeline Systems Group
ESO Pipeline Workshop, Oct 24 2014
Reflex Canvas
ESO Pipeline Workshop, Oct 24 2014
Workflow Overview
ESO Pipeline Workshop, Oct 24 2014
Calibration cascade
ESO Pipeline Workshop, Oct 24 2014
Designing a workflow● Steps previous to use Reflex:
● Design modular pipelines: i.e. with recipes performing granular tasks. Avoid cases where I/O of the recipes depend on recipe parameters.
● Think about the supported observing modes. Optional calibrations sometimes imply quite different reduction cascades.
● Start with an stable version of the pipeline I/O and parameters.● Carefully design the FITS file categories.● Design the workflow layout in paper
● Steps using Reflex tool:● Create Data Organisation rules (OCA rules) which mimic the workflow
layout as closely as possible● Use a workflow template (iiinstrument)● Create the graphical workflow layout● Skip interactivity development until the end.
ESO Pipeline Workshop, Oct 24 2014
Documentation
● Reflex User Manual ftp://ftp.eso.org/pub/dfs/reflex/ReflexUserManual-3.5.pdf
● Reflex Workflow Developers Guide: ftp://ftp.eso.org/pub/dfs/reflex/reflex_dev_guide-1.0.pdf
●Reflex paper: http://eso.org/sci/software/pipelines/documents/reflex_aa.pdf
ESO Pipeline Workshop, Oct 24 2014
Data Organisation: OCA Rules
● OCA rules are the mechanism used to perform Data organisation in a Reflex workflow● OCA rules and workflow graphical layout are very closely related.● OCA rules are written in a text file with a given syntax.● There are three types of rules:
● Classification. Based on FITS keywords, it specifies the category of the file we have (“I am a raw flat”)
● Organization. Files are grouped by keywords and each group triggers an action (“These flats create a master flat”)
● Association. Each action can associate static calibrations or products created by other actions (“To create a master flat I need a master bias“)
ESO Pipeline Workshop, Oct 24 2014
File Purpose
● Category is not enough to determine the use of a file.
Example: bias for a science can be different as bias for the flat● The Purpose is the role a file will have in the reduction cascade => Path in the association tree● A given file can have a list of purposes.
ESO Pipeline Workshop, Oct 24 2014
Reflex Actors: FitsRouter
● The FitsRouter will split the input by
category.There are two modes:● Routing by category explicitly. Just create a port
with the name of the category.● Configuring a port. Create a configuration
parameter with the name PORT_config and list the desired categories.
ESO Pipeline Workshop, Oct 24 2014
Reflex Actors: DataSetChooser● It allows to select the Datasets to reduce● Datasets can be inspected and the calibration cascade will be shown.● Purposes can be seen as the path to the leaf in the tree● If a Dataset is incomplete, it will show which are the missing files.
ESO Pipeline Workshop, Oct 24 2014
Reflex Actors: RecipeExecuter
● The RecipeExecuter has to be instantiated explicitly:
Tools -> Instantiate Component. ● The list of recipes that will appear there is the result of the command esorex –recipes.● For each recipe parameters, an actor parameter is created with the name “recipe_param_nn”, where nn corresponds to the parameter order. If recipe parameters change, the RecipeExecuter should be reinstantiated.
ESO Pipeline Workshop, Oct 24 2014
Reflex Actors: SofSplitter & SofAcummulator
● In order to handle properly Purposes, the RecipeExecuter's have to be enclosed between SofSplitter/SofAccumulator.● SofSplitter will create as many groups as different purposes are in the input and will trigger the RecipeExecuter for each of the purpose groups.● SofAccumulator simply combines back all the products in a single channel.
ESO Pipeline Workshop, Oct 24 2014
Reflex Actors: ProvenanceExplorer
It displays the effective reduction cascade used to create a given product. It will be renamed to ProductExplorer
ESO Pipeline Workshop, Oct 24 2014
RecipeLooper
ESO Pipeline Workshop, Oct 24 2014
PythonActor I
● It is able to execute generic python code.● To translate from/to Reflex ports to/from python script arguments a special syntax is used, with the help of a Python module● To create a python actor, use the menu Tools->Instantiate Component and type org.eso.PythonActor.
ESO Pipeline Workshop, Oct 24 2014
Python Actor II
A python script can be “reflexed” using the following syntax:from reflex import *
parser = ReflexIOParser()
#Define inputs
parser.add_option("-i", "--in_sof", dest="in_sof")
#Define outputs
parser.add_output("-o", "--out_sof", dest="out_sof")
(inputs, args) = parser.parse_args()
outputs = parser.get_outputs()
#Set the output
outputs.out_sof = inputs.in_sof
parser.print_outputs()
sys.exit()
Importing Reflex
Define Inputs/Outputs
Getting inputs
Setting outputs
ESO Pipeline Workshop, Oct 24 2014
Reflex Python Framework I
● Reflex is delivered with a python library that helps to create interactive user interface rather easy using matplotlib and wxPython.● The reflex_interactive_app module is a framework to create interactive windows.● The pipeline_product module eases the reading of FITS images, spectra and tables, using pyfits.● The pipeline_display module produces scatter plots, image displays and spectra plots using matplotlib.● The reflex_plot_widgets module allows to add extra interactivity within the matplotlib plots.● These modules are accessible directly when using a python script inside the PythonActor.
ESO Pipeline Workshop, Oct 24 2014
Examples of interactive windows
ESO Pipeline Workshop, Oct 24 2014
Examples of interactive windows
ESO Pipeline Workshop, Oct 24 2014
Reflex Python Framework II
Example of using reflex_interactive_app framework: def setInteractiveParameters(self):
return [reflex.RecipeParameter(recipe=rec_name, displayName="par1", group="Limits"]
def readFitsData(self, fitsFiles):
self.frames = dict()
for f in fitsFiles:
self.frames[f.category] = PipelineProduct(f)
pro_raw = self.frames[“MY_CAT”]
pro_raw.readImage()
def addSubplots(self, figure):
self.raw_plot = figure.add_subplot(111)
def plotProductsGraphics(self):
img_raw_disp = pipeline_display.ImageDisplay()
img_raw_disp.setLabels('X', 'Y')
tooltip = paragraph("Raw image")
img_raw_disp.display(self.raw_plot, "Raw image", tooltip, pro_raw.image)
Defining the recipeparameters to show
Reading the FITSData from the input
Defining the plotlayout
Plotting the data
ESO Pipeline Workshop, Oct 24 2014
Directories setup● There are 6 directories:
– CALIB_DATA_DIR: Extra directory with static calibrations provided by the pipeline. Used by the DataOrganizer
– RAWDATA_DIR: The dir where the user data resides. Used by the DataOrganizer
– BOOKKEEPING_DIR: Reflex stores here all the information about previous executions (I/O sof, parameters, etc...). Used by the DataOrganizer, RecipeExecuter, DataSetChooser, PythonActor and ProvenanceExplorer
– LOGS_DIR: The output of esorex are stored here. Used by the RecipeExecuter.– TMP_PRODUCTS_DIR: The products created by each execution of a recipe are
saved here. Used by the RecipeExecuter– END_PRODUCTS_DIR: The science products from each dataset are stored
here. Used by the ProductRenamer, ProvenanceExplorer● The installation script of Reflex takes care of setting up all the workflow directories for the demo data● Workflows usually setup a ROOT_DATA_DIR and define all the other dirs from this one
ESO Pipeline Workshop, Oct 24 2014
Saving workflows
● There are two formats to save a workflow:● KAR. This is the default format.
Binary format● XML. This has to be exported.
“Human-readable”● The paths to the python scripts and
OCA rules file are saved together with the workflow.
ESO Pipeline Workshop, Oct 24 2014
Lazy Mode● Lazy mode for RecipeExecuter.
– It works by comparing the input of the current execution with all the previous recipe executions:
• All files must be the same
• All files must have the same checksum
• All files must have the same date
• All recipe parameters must be the same
– If a recipe at the beginning of the workflow is set to Not-Lazy mode, the input of the next recipes will be new and lazy mode will not be triggered.
● Lazy mode for DataOrganizer.– It avoids the organization of all the data in subsequent workflow
runs. – It works similar to lazy mode for recipes
ESO Pipeline Workshop, Oct 24 2014
Questions?
ESO Pipeline Workshop, Oct 24 2014
Backup slides...
ESO Pipeline Workshop, Oct 24 2014
Data Organisation: OCA Rules II
Classification:
if DPR.CATG like "%SCIENCE%" and DPR.TYPE like "%OBJECT%" then
{
REFLEX.CATG = "RRRECIPE_DOCATG_RAW";
REFLEX.TARGET = "T";
}
if DPR.CATG like "%CALIB%" and DPR.TECH like "%IMAGE%"
and DPR.TYPE like "%STD%" then
{
REFLEX.CATG = "RRRECIPE_CALIB_DOCATG_RAW";
}
ESO Pipeline Workshop, Oct 24 2014
Data Organisation: OCA Rules III
Organisation:
select execute(CALIB_IMG) from inputFiles where REFLEX.CATG == “RAW”
group by INS.FILT1.NAME, OBS.ID, OBS.TARG.NAME, TPL.START;
ESO Pipeline Workshop, Oct 24 2014
Data Organisation: OCA Rules IV
Association:
action CALIB_IMG
{
minRet = 0; maxRet = 1;
select file as STATIC_MASK from calibFiles where REFLEX.CATG == "STATIC_MASK";
recipe rrrecipe_calib;
product IMG_CALIBRATED { REFLEX.CATG = "IMG_CALIBRATED"; PRO.CATG = "IMG_CALIBRATED"; PRO.EXT="tpl_0000.fits";}
}
action COMBINE_IMG
{
minRet = 1; maxRet = 1;
select file as IMG_CALIBRATED from calibFiles where PRO.CATG == "IMG_CALIBRATED";
recipe rrrecipe;
product IMG_OBJ_COMBINED { PRO.CATG = "IMG_OBJ_COMBINED";PRO.EXT="tpl_0001.fits";}
}
ESO Pipeline Workshop, Oct 24 2014
How to configure an actor
● Each actor has input and output ports.● To edit them, right click on “Configure ports”
● An input multiport can receive several tokens which are combined together● An output multiport distributes the token to several actors● Many output ports are single ports and cannot be changed.● Each actor has some parameters that can be configured by right click “Configure actor”.
ESO Pipeline Workshop, Oct 24 2014
Reflex Actors: ProductRenamer
● Rename_keywords specifies the pattern used to create the file name, based on the available FITS keywords.
● The directory to be used will be FinalProductDirectory/SubDir. Subdir is usually defined by the workflow to be the execution timestamp.
● The actor is actually a jython script which can be modified by double-clicking it. Its code is part of its configuration, and therefore it is saved with the workflow. To use a more recent version, just copy and paste.
ESO Pipeline Workshop, Oct 24 2014
Python Actor Example
An example with some data processing: #Retrieve input
in_sof = inputs.in_sof
#Get the input files
files, dataset = parseSof(in_sof)
#Do the stuff
for file in files:
hdulist = pyfits.open(file.name,mode='update')
newdata = hdulist[0].data / 2
hdulist[0].data = newdata
hdulist.flush()
Parsing the Reflex format
Using pyfits
Updating the file
ESO Pipeline Workshop, Oct 24 2014
Saving workflows in SVN●.● If you are going to save the workflow in SVN, eport as KML and do the following:
● Set ROOT_DATA_DIR to ROOT_DATA_PATH_TO_REPLACE
● Set CALIB_DATA_DIR to 1CALIB_DATA_PATH_TO_REPLACE/ins-@VERSION@
● Set the paths to OCA rules to @prefix@/share/esopipes/ins-@VERSION@/reflex/ins_wkf.oca
● Set the paths to Python actors to @prefix@/share/esopipes/ins-@VERSION@/reflex/ins_interact.py