Pipeline Day - ESO · 1 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014 Pipeline Day 9:00...
19
1 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014 Pipeline Day 9:00 Welcome and Introduction (P.Ballester) 9:30 The ESO Common Pipeline Library (R.Palsa) 10:00 The High-Level Data Reduction Library (W.Freudling, A.Gabasch) 10:30 Coffee break 10:45 The ESO-Reflex Environment (W.Freudling, E.Garcia) 11:15 Pipeline Development and Testing (A.Szostak) 11:30 Discussion: Maintenance, new Libraries, Interfaces 12:30 Lunch break 14:00 Development Live Demo (E.Garcia, R.Palsa, J.Taylor) 16:00 Coffee Break / End of Workshop
Transcript of Pipeline Day - ESO · 1 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014 Pipeline Day 9:00...
1 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Pipeline Day
9:00 Welcome and Introduction (P.Ballester)
9:30 The ESO Common Pipeline Library (R.Palsa)
10:00 The High-Level Data Reduction Library (W.Freudling, A.Gabasch)
10:30 Coffee break
10:45 The ESO-Reflex Environment (W.Freudling, E.Garcia)
11:15 Pipeline Development and Testing (A.Szostak)
11:30 Discussion: Maintenance, new Libraries, Interfaces
12:30 Lunch break
14:00 Development Live Demo (E.Garcia, R.Palsa, J.Taylor)
16:00 Coffee Break / End of Workshop
2 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Science Operations Software Department
Science Operations Software Department
(P.Ballester)
Data Flow Infrastructure Group
(S.Zampieri)
Pipeline Systems Group
(S.Castro)
Software Engineering Group
(M.Zamparelli)
A. Gabasch E. Garcia J. Gonzalez Y. Jung L. Lundin A. Modigliani R. Palsa J. Vinther
CFTE: K. Shabun A. Szostak J. Taylor
3 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Data Flow System Deliverables and Integration
4 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Data reduction environments
• Observatory Pipeline • On-the-fly data processing (event driven) • Template-based processing • Static calibration database (only certified products are used)
• Quality Control Pipeline • Batch processing of complete data sets (all science and calibration data produced
by one ESO instrument in one night) • Generation of master calibration frames
• Science Data Products Pipeline • Batch processing of science archive with best calibrations and fixed parameters
• Desktop science-grade data reduction • Modular and additional recipes are available • Interactive windows
5 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Observatory Pipeline
Instrument Raw Data
Archive
PIPELINE OFFLINE
Further Analysis
Raw data
Processed data
On-Line Archive Sytem
Shipping
6 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Quality Control: Information by Instrument
Similar pages for every VLT/VLTI instrument
7 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Phase 3 Internal Data Products
• IDP: Internal Data Products • Data products produced by ESO for the
community • Standard formats for: Imaging, 1D Spectra, 3D IFU
Data, Interferometry Data
• Complementary to the Phase 3 process of ingestion of External Data Products in the Science Archive
• Currently implemented for the UVES, X-SHOOTER, GIRAFFE pipelines (in-house versions).
• Extremely popular archive data products • To be added to all coming pipelines • Will be supported by CPL/HDRL functions
8 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Reflex Workflows
9 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Public Releases
• www.eso.org/pipelines
• Linked from all ESO instrument pages
• Single-entry point to all
relevant documentation, packages, and demonstration data
• Links to FAQ, Data Reduction Forum, Articles.
13
10 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
ETC/ Observation Preparation Tools
11 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Data Reduction Library
PSG
Paranal Science Operations
INS Commissioning Team
Data Flow Operations
DICB/Archive
Public Release
User Community
12 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
1618: History
• 1999: 1618 version 1.0 • MIDAS based C,F77,C++ or Data Flow C/C++ classes • Data Reduction Specification & Procedures • Identifies a minimum of 8 pre-defined deliverables
• 2004: 1618 version 2.0 • ANSI-C ESO-CPL: Common Pipeline Library • DRL Specifications, Detailed Design, Code • Identifies a minimum of 19 pre-defined deliverables
• 2011: 1618 version 3.0 • Removed DFS Impact Document • Added science-grade pipelines, Reflex, acceptance procedures • Deliverables, ESO internal workflow, tighter validation
• Next version: • For next VLT instruments and E-ELT • Planned additions: HDRL, IDPs introduced in the course of projects
13 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
High Level Data Reduction Library
• HDRL: High-Level Data Reduction Library • Provides well tested, optimised algorithms for processing
steps that are shared by pipelines • Developed in collaboration between Pipeline Systems
and Science Data Product groups • Development standard for future pipelines • See presentation later this morning
14 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Code Validation
• Usage of CPL recipe template
• Following CPL coding standard • Usage of external libraries • Namespace protection
• Execution Tests • Completeness of the set of
recipes and DRL functions • Availability/representativity of test
data • Proper execution of recipes • Generation of products • Memory leaks • Unit tests • Documentation
• Detailed Validation • Correctness of results • Validation of input • FITS compliance • User-friendly documentation • Data reduction cascade • Unit tests
• Performance and Portability
• Execution speed • Standard platforms: Scientific
Linux
15 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Scientific Validation
• Science Grade Data Products are independent of the science goals
• Products calibrated in physical units with error estimates • No residual systematic error • S/N close to the achievable optimum (e.g. ETC
prediction) • Validation involving Science Data Products group
16 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
From Phase A to FDR
• Phase A and PDR Preparation • Reuse existing documents as templates • Identify extra needs: observation preparation, visualisation, link to the data analysis?
• Learning CPL before the FDR • First recipes in CPL, coding standards, memory management, … • Review existing pipelines, develop prototypes • Coding starts officially only after the FDR
• FDR Preparation • Prototype data reduction algorithms (test and simulated data) • Structure of data products (for QC, IDP, Workflow usage, headers and structures) • Validation and Test Plan • Observatory Pipeline is template based
— Data combination, interactive processing — Further processing requirements
17 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
From FDR to Science Verification
• FDR to PAE • Plan more validation effort for the first recipes, • In general, plan enough time for testing and finalising • Progress meetings vevery 6 to 8 weeks • Regular intermediate software releases (3 to 4 months), including test reports • Keep improving simulated or adapted data, in sync with instrument schedule
• PAE • Plan for the complete set of recipes to be ready at PAE • Validated data reduction algorithms (using laboratory and simulated data) • Have ready a few alternative calibration/reduction methods
• Commissioning to Science Verification • Test and validate on instrument and sky data • Identify and solve the unexpected problems • Instrument cannot be accepted without the pipeline being accepted
18 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Pipeline Maintenance
19 - P.Ballester – ESO Pipeline Workshop - 24 Oct 2014
Pipeline Day
9:00 Welcome and Introduction (P.Ballester)
9:30 The ESO Common Pipeline Library (R.Palsa)
10:00 The High-Level Data Reduction Library (W.Freudling, A.Gabasch)
10:30 Coffee break
10:45 The ESO-Reflex Environment (W.Freudling, E.Garcia)
11:15 Pipeline Development and Testing (A.Szostak)
11:30 Discussion: Maintenance, new Libraries, Interfaces
12:30 Lunch break
14:00 Development Live Demo (E.Garcia, R.Palsa, J.Taylor)
16:00 Coffee Break / End of Workshop
