1 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Pipeline Day

9:00 Welcome and Introduction (P.Ballester)

9:30 The ESO Common Pipeline Library (R.Palsa)

10:00 The High-Level Data Reduction Library (W.Freudling, A.Gabasch)

10:30 Coffee break

10:45 The ESO-Reflex Environment (W.Freudling, E.Garcia)

11:15 Pipeline Development and Testing (A.Szostak)

11:30 Discussion: Maintenance, new Libraries, Interfaces

12:30 Lunch break

14:00 Development Live Demo (E.Garcia, R.Palsa, J.Taylor)

16:00 Coffee Break / End of Workshop

2 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Science Operations Software Department

Science Operations Software Department

(P.Ballester)

Data Flow Infrastructure Group

(S.Zampieri)

Pipeline Systems Group

(S.Castro)

Software Engineering Group

(M.Zamparelli)

A. Gabasch E. Garcia J. Gonzalez Y. Jung L. Lundin A. Modigliani R. Palsa J. Vinther

CFTE: K. Shabun A. Szostak J. Taylor

3 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Data Flow System Deliverables and Integration

4 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Data reduction environments

•   Observatory Pipeline •   On-the-fly data processing (event driven) •   Template-based processing •   Static calibration database (only certified products are used)

•   Quality Control Pipeline •   Batch processing of complete data sets (all science and calibration data produced

by one ESO instrument in one night) •   Generation of master calibration frames

•   Science Data Products Pipeline •   Batch processing of science archive with best calibrations and fixed parameters

•   Desktop science-grade data reduction •   Modular and additional recipes are available •   Interactive windows

5 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Observatory Pipeline

Instrument Raw Data

Archive

PIPELINE OFFLINE

Further Analysis

Raw data

Processed data

On-Line Archive Sytem

Shipping

6 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Quality Control: Information by Instrument

Similar pages for every VLT/VLTI instrument

7 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Phase 3 Internal Data Products

•   IDP: Internal Data Products •   Data products produced by ESO for the

community •   Standard formats for: Imaging, 1D Spectra, 3D IFU

Data, Interferometry Data

•   Complementary to the Phase 3 process of ingestion of External Data Products in the Science Archive

•   Currently implemented for the UVES, X-SHOOTER, GIRAFFE pipelines (in-house versions).

•   Extremely popular archive data products •   To be added to all coming pipelines •   Will be supported by CPL/HDRL functions

8 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Reflex Workflows

9 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Public Releases

•   www.eso.org/pipelines

•   Linked from all ESO instrument pages

•   Single-entry point to all

relevant documentation, packages, and demonstration data

•   Links to FAQ, Data Reduction Forum, Articles.

13

10 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

ETC/ Observation Preparation Tools

11 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Data Reduction Library

PSG

Paranal Science Operations

INS Commissioning Team

Data Flow Operations

DICB/Archive

Public Release

User Community

12 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

1618: History

•   1999: 1618 version 1.0 •   MIDAS based C,F77,C++ or Data Flow C/C++ classes •   Data Reduction Specification & Procedures •   Identifies a minimum of 8 pre-defined deliverables

•   2004: 1618 version 2.0 •   ANSI-C ESO-CPL: Common Pipeline Library •   DRL Specifications, Detailed Design, Code •   Identifies a minimum of 19 pre-defined deliverables

•   2011: 1618 version 3.0 •   Removed DFS Impact Document •   Added science-grade pipelines, Reflex, acceptance procedures •   Deliverables, ESO internal workflow, tighter validation

•   Next version: •   For next VLT instruments and E-ELT •   Planned additions: HDRL, IDPs introduced in the course of projects

13 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

High Level Data Reduction Library

•   HDRL: High-Level Data Reduction Library •   Provides well tested, optimised algorithms for processing

steps that are shared by pipelines •   Developed in collaboration between Pipeline Systems

and Science Data Product groups •   Development standard for future pipelines •   See presentation later this morning

14 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Code Validation

•   Usage of CPL recipe template

•   Following CPL coding standard •   Usage of external libraries •   Namespace protection

•   Execution Tests •   Completeness of the set of

recipes and DRL functions •   Availability/representativity of test

data •   Proper execution of recipes •   Generation of products •   Memory leaks •   Unit tests •   Documentation

•   Detailed Validation •   Correctness of results •   Validation of input •   FITS compliance •   User-friendly documentation •   Data reduction cascade •   Unit tests

•   Performance and Portability

•   Execution speed •   Standard platforms: Scientific

Linux

15 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Scientific Validation

•   Science Grade Data Products are independent of the science goals

•   Products calibrated in physical units with error estimates •   No residual systematic error •   S/N close to the achievable optimum (e.g. ETC

prediction) •   Validation involving Science Data Products group

16 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

From Phase A to FDR

•   Phase A and PDR Preparation •   Reuse existing documents as templates •   Identify extra needs: observation preparation, visualisation, link to the data analysis?

•   Learning CPL before the FDR •   First recipes in CPL, coding standards, memory management, … •   Review existing pipelines, develop prototypes •   Coding starts officially only after the FDR

•   FDR Preparation •   Prototype data reduction algorithms (test and simulated data) •   Structure of data products (for QC, IDP, Workflow usage, headers and structures) •   Validation and Test Plan •   Observatory Pipeline is template based

—  Data combination, interactive processing —  Further processing requirements

17 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

From FDR to Science Verification

•   FDR to PAE •   Plan more validation effort for the first recipes, •   In general, plan enough time for testing and finalising •   Progress meetings vevery 6 to 8 weeks •   Regular intermediate software releases (3 to 4 months), including test reports •   Keep improving simulated or adapted data, in sync with instrument schedule

•   PAE •   Plan for the complete set of recipes to be ready at PAE •   Validated data reduction algorithms (using laboratory and simulated data) •   Have ready a few alternative calibration/reduction methods

•   Commissioning to Science Verification •   Test and validate on instrument and sky data •   Identify and solve the unexpected problems •   Instrument cannot be accepted without the pipeline being accepted

18 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Pipeline Maintenance

19 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014

Pipeline Day

9:00 Welcome and Introduction (P.Ballester)

9:30 The ESO Common Pipeline Library (R.Palsa)

10:00 The High-Level Data Reduction Library (W.Freudling, A.Gabasch)

10:30 Coffee break

10:45 The ESO-Reflex Environment (W.Freudling, E.Garcia)

11:15 Pipeline Development and Testing (A.Szostak)

11:30 Discussion: Maintenance, new Libraries, Interfaces

12:30 Lunch break

14:00 Development Live Demo (E.Garcia, R.Palsa, J.Taylor)

16:00 Coffee Break / End of Workshop