5-1 DMS Data Products Nov. 6, 2013 S&OC Data Management System Design Review 3.

5-1

DMS Data Products

Nov. 6, 2013 S&OC Data Management System Design Review 3

5-2

Presentation Outline

Data Definition Review

NIRSpec MSA Metadata File

Imaging Associations

Future Data Product Design


5-3

Data Product References

DMS Requirements Document (JWST-STScI-002249, JWST-RQMT-002941)

Generation of Level 1a Science Data Products (JWST-STScI-002566)

DMS Level 1 & 2 Data Product Design (JWST-STScI-002111)

S&OC Subsystem IRCD (JWST-IRCD-013128, JWST-STScI-000905) Contains specification for data exchanged between S&OC

subsystems

PRDS to Users IRCD Volume III: S&OC Subsystems (JWST-STScI-000949) Contains specification for S&OC data contained in PRD

Definition of the Flexible Image Transport System (FITS), version 3.0, A&A 524, A42 (2010) FITS standard


5-4

Data Level Definition

Science Data Archive

Object Catalog

Level 0: Science telemetry packet files

Level 1a: Original FITS files

Level 1b: Uncalibrated FITS files

Level 2a: Ramp calibrated exposure FITS files

Level 2b: Fully calibrated exposure FITS files

Level 4: Products of data analysis

Extractedobjects

Proposer

Level 3: Calibrated, associated FITS files


5-5

Exposure File Naming Convention

At data levels 1 and 2, the science data files are FITS format files that contain the pixel values for a single exposure from a single detector.

Goals for file name: DMS data user is able to map the data file to the observation input through the

Astronomer's Proposal Tool (APT). Operations staff can map every exposure to a construct in the JWST commanding scripts

that was used to make the observation request.

The file name is based on the following exposure hierarchy: Program (PPPPP) Observation (OOO) Visit (VVV) Visit Group (GG) Parallel Sequence id (S, 1 prime, 2-5 parallel) Activity (AA, base 36) Exposure id (EEEEE)

Example: jw<PPPPP><OOO><VVV>_<GGSAA>_<EEEEE>_<detector >_<suffix> .fits jw12345012009_01101_00012_nrcalong_uncal.fits

Visit ID explicitly embedded in exposure file name

Exposure number resets with each new activity The exposure number within the file name does not necessarily monotonically increase for each

exposure within a single visit.Nov. 6, 2013 S&OC Data Management System Design Review 3

5-6

NIRSpec MSA


5-7

Relevant DMS Requirements


Req. # Requirement SOC parent

DMS-234 3.3.2.1 Science Data Product GenerationThe DMS shall generate Science Data Products from Science Data.

3.4.5.1.3 Science Data Product Generation [SOC-529]

DMS-133 5.1.6.1 Include PPS Information in Header Keyword ValuesSDP shall populate the FITS data headers with proposal, planning, and scheduling information received from PPS.

3.4.2.1.2 Level 1 Processing [SOC-1689]

DMS-192 6.2.1.1 Generate Science Data ProductsThe DMS shall generate Calibrated Science Data Products from Science Data for all supported modes of the JWST science instruments, as defined in section 3.1.3 of the S&OC Requirements Document (JWST-RQMT-002032, JWST-STScI-000046) under Science Instrument Capabilities.

3.4.3.1.2 Science Data Calibration [SOC-513]

DMS-460 6.2.1.4 Create Level-2 Data ProductsThe Calibration Software shall create Level 2 Science Data Products as specified in the User Handbooks, OP-07.


DMS-461 6.2.1.5 Create Level-3 Data ProductsThe Calibration Software shall create Level 3 Science Data Products as specified in the User Handbooks, OP-07.


DMS-480 6.2.2.17 Spectra CombinationThe Calibration software shall have the capability of combining spectra extracted from different exposures.


DMS-481 6.2.2.18 Background SubtractionThe Calibration software shall have the capability of subtracting the background from a spectrum.


DMS-482 6.2.2.19 Spectroscopic ExtractionThe Calibration software shall have the capability of extracting a 1-d spectrum from 2-d spectroscopic data.


DMS-549 6.3.1.1 Association DefinitionThe DMS shall define Associations that consist of exposures that may be combined during calibration.


DMS-553 6.3.1.5 Contemporaneous Calibration AssociationsThe DMS shall associate contemporaneous calibration exposures with the relevant science exposures.


5-8

NIRSpec MSA Reference Documentation

“NIRSpec Data Associations Study: Associating MSA Spectral Data in the Calibration Pipeline and MSA Archive File Structures,” T Beck, J Valenti, J Muzerolle, T Keyes, JWST-STScI-003366, May 24, 2013 Association product study led to definition of data products

“Further Definition of the NIRSpec Science Templates,” D. Karakla, T. Beck, W. Blair, C. Keyes, J. Muzerolle, K. Pontoppidan, D. Soderblom, J. Tumlinson, and J. Valenti, JWST-STScI-002288, 18 March 2011.

“STScI NIRSpec Calibration Pipeline Processing Description – Rev. A,” Tracy Beck, JWST-STScI-001859, 15 June 2011.

NIRSpec MSA Planning Tool “Optimization Strategies for the NIRSpec MSA Planning Tool,” James

Muzerolle, JWST-STScI-001851, 9 December 2009. “MSA Planning Tool – A First Algorithm to Plan an Observation,” Diane

Karakla and the MSA Tool Algorithm Working Group, JWST-STScI-003003, 22 October 2012.

“MSA Planning Tool Design Requirements,” Diane Karakla and the MSA Tool team, JWST-STScI-002971, 4 December 2012.


5-9

NIRSpec MSA Design Summary

Data files Level 1 and 2 science data files based on exposure/detector Level 3 science data files organized by sources DMS metadata file captures MSA metadata from PPS database

Archive paradigm change Save only least combined and most combined data for a program Least combined data – individual exposure data products at level 1

and 2

- No pixel resampling

Most combined data – level 3 data with the most sources

- Changes over time as more visits execute

- New combined products should always be generated using the exposure-level, least combined products as the inputs

- New combined products replace previous versions in the archive


5-10

MSA Data Flow


Data Processing

Proposal Input

MSA Pre-

image

MSA Planning

Tool

PPS Database

APT

Science Data

Processing

Recorded Science Data File

Level 1b Science Data File

Level 2 Calibration

Level 2b Science Data File

Level 3 Calibration

Level 3 Science Data File

MSA Metadata

File

JWSTDetector/

exposure based

Source based

PI

Source Catalog

5-11S&OC Data Management System Design Review 3

NIRSpec MSA Glossary (1/2)

Pre-image – high quality image provided by PI in FITS format used to

select sources for MSA observations relative accuracy of target coordinates from the pre-image must be

known to within 5 mas

Target – position to point the telescope (V2,V3) with fixed orientation

Reference stars – set of 8-20 reference stars to provide relative

astrometry for target acquisition

Catalog – set of sources used to plan observation Not all sources may be observed

Sources – objects of observation in MSA shutter

Target set – set of objects observed simultaneously Target set needs to prevent overlapping spectra for any given

configuration

Nov. 6, 2013


NIRSpec MSA Glossary (2/2)

Slitlet – set of related micro-shutters For example, one shutter contains the source, other

shutters in the same column contain local background

For medium and high resolution gratings, only one open shutter per row

- Prisms will not spread spectra as far as medium and

high resolution gratings. In that case, multiple

shutters can be open per row without overlapping

spectra.

Rows with a failed open shutter are blocked from use

Rows with a failed closed shutter are tolerated as long as the failed closed shutter is not part of the slitlet

Sources can be dithered within the slitlet by small angle maneuvers- Improves accuracy of background subtraction

- Dithers the source spectrum on the detector between

sub-exposures so that bad detector pixels can be

corrected

Nod – small dither to move source position within

slitlet without MSA reconfigurationNov. 6, 2013


NIRSpec MSA – DMS Metadata File

Topics to be covered: How DMS/SDP extracts MSA metadata from PPS database Additional MSA information captured in calibration reference files DMS MSA metadata data product design

DMS MSA Metadata File Science Data Processing component extracts information from PPS

and generates the MSA Metadata File MSA Metadata File used by archive users and calibration software

- Archive users do not have access to PPS database

- Provides interface to PPS database for calibration software

Nov. 6, 2013


NIRSpec MSA Metadata Information Flow

Nov. 6, 2013

SDP

PPS Database

NIRSpec MSA Metadata Product

Calibration

MSA Planning Tool

APT

CRDS

Stuck Open/Closed Shutter

Reference File

MSA to Detector Distortion

Reference File

NIRSpec MSA

configuration info

NIRSpec MSA

PI input

Observ-ation

parameters

Archive

Archive User


DMS Relevant PPS Tables

Nov. 6, 2013

Observatory Status File:

programobservationvisitpointing_sequence_number

PPS database

nirspec_msa_at_pointing

programobservationvisitpointing_sequence_number

programmsa_config_id

msa_default_state

programmsa_metadata_id

nirspec_msa_config


shutter_quadrantshutter_rowshutter_column

nirspec_target_acq


observationvisitorder_number

reference_star_xreference_star_yreference_star_flux

programsource_id

nirspec_sources

programsource_id

aliascatalog_idradecpre_image_id

nirspec_msa_metadata

programmsa_metadata_id

programsource_id

slitlet_idshutter_quadrantshutter_rowshutter_columnshutter_statebackgroundestimated_source_in_shutter_xestimated_source_in_shutter_y


Comments on PPSDB Tables

nirspec_msa_at_pointing Allows link between Observatory Status File and NIRSpec MSA PPS database tables msa_default_state = ‘open’ or ‘closed’

nirspec_msa_config Lists shutters not in default state Configuration id unique within a program

nirspec_msa_metadata DMS specific information msa_metadata_id unique within a program Information on shutter level slitlet_id can apply to multiple shutters slitlet_id unique within program shutter_state =‘open’ or ‘closed’: Can force shutters closed to mask bright source background = T or F background shutter can be assigned to multiple sources

PPS to compute source PSF position in shutter in x and y directions of NIRSpec ideal coordinate frame in units of fraction of

shutter length and width Link to nirspec_sources for source information

nirspec_target_acq New visit and target acq required for configuration change Program, observation, visit, and order_number can be used to link to exposure specification 8-20 reference stars per target acq. Link to nirspec_sources for source information

nirspec_sources pre_image_id unique within a program Map pointing to pre-image

Nov. 6, 2013


PPS nirspec_msa_metadata Table Details

description field data type

values

Metadata id msa_metadata_id I4 unique within a program

Row for the open shutter shutter_row I4 1..365

Column for the shutter shutter_column I4 1..171

Quadrant of the shutter shutter_quadrant I2 1..4

Shutter is opened or closed shutter_state VC(6) Open/Closed

Source associated with this shutter source_id I4 references a source

Is this background background char(1) Y/N

Identifier to indicate the logical group of shutters to which this shutter belongs.

slitlet_id I4 unique within a program

Source PSF position as offset from the shutter center in NIRSpec ideal coordinate frame x direction

estimated_ source_in _ shutter_x

R8 fraction of shutter size in x

Source PSF position as offset from the shutter center in NIRSpec ideal coordinate frame y direction

estimated_ source_in _ shutter_y

R8 fraction of shutter size in y

Nov. 6, 2013


PPS nirspec_sources Table

Nov. 6, 2013

Field name Type Description Units/range

program I4 Program number 1…99999

source_id I4 Unique ID of a MSA source within a program

alias VC31 Optional name that the observer may give to a source

catalog_id VC20 PI catalog from which the source was taken

ra R8 Coordinates of source in right ascension

degrees

dec R8 Coordinates of source in declination degrees

pre_image_id VC20 Name of the pre-image that was used to measure position for this source


Additional Files Input to Calibration

Shutter stuck open/closed file(s) and MSA to detector map to be delivered to CRDS through NIRSpec Instrument Team as reference files Information not provided in MSA Metadata File

File(s) to indicate shutters stuck open or closed Stuck shutters may change between time of phase II proposal prep and

visit execution Also provides contrast – fraction of light to get through shutter (even

when shutter open or closed)

Distortion maps Needed for distortion correction in MSA plane

- Not a pixel map, polynomials will be used to characterize distortion

- Distortion in detector plane characterized in SIAF

Used for on-board target acquisition Needed for DMS calibration Same at any given point in time regardless of aperture The mapping file could change from the time of planning to the time of

execution

Nov. 6, 2013


Science Exposure Header Keyword Information

Pointer to pre-image file name One pre-image per configuration A pre-image can generate multiple configurations

Pointer to stuck open/closed CRDS reference file

Pointer to MSA to detector map CRDS reference file

Pointer to MSA metadata file

Information to map exposure to configuration and metadata Program Observation Visit Pointing sequence number MSA configuration id MSA metadata id MSA default state

Engineering parameter needed to provide grating wheel positionNov. 6, 2013


Exposure to ID Correspondence

Each exposure has a single configuration id and a single metadata id

Level 1 and 2 files for detectors NRS1 and NRS2 use same configuration id and metadata id

A configuration id can be applied to multiple exposures For example, exposures through multiple gratings with same

configuration

There can be multiple metadata ids within configuration For example, background shutter changes between nods

A metadata id can be applied to multiple exposures, but will typically be applied to a single pointing For example, exposures through multiple gratings

Nov. 6, 2013


NIRSpec MSA Metadata Data Product

Generated by SDP (Level 1 processing)

Includes information from planning database

FITS format file

One file for each unique MSA configuration Uniqueness defined by program, msa_config_id File name format: jw<visit_id>_<msa_config_id>_msa.fits

- Visit_id 11 characters: <ppppp><ooo><vvv>

- Msa_config_id: 2 base-36 characters

File needs to be associated with all exposures and higher level data products using that configuration Multiple metadata files per data product

Metadata data product generated at time of first exposure using that configuration

Nov. 6, 2013


FITS File Structure

Primary Header

Source Info Extension

Extension Header

Extension Data Table

Shutter State Extension

Extension Header

Extension DataShutter Info Extension

Extension Header

Extension Data Table

Nov. 6, 2013


Shutter State Data

Data array to show anticipated state of all shutters Planned shutter state based on PPS database information Calibration can update shutter state information from reference files

and confirmation images

FITS data array values Closed - 0, Open - slitlet_id If shutter open without being assigned a slitlet_id, set data array

value = -1. Imaging mode settings TBD

Nov. 6, 2013


Shutter Info Extension

Table listing all shutters included in a slitlet Fields would include:

Mimics PPS nirspec_msa_metadata table. Assumes all useful shutters assigned a slitlet_id Background shutters or slitlets can be assigned to multiple sources

- Quadrant, row, and column do not necessarily define a unique table row

metadata_id

shutter_ quadrant

shutter_ row

shutter_column

slitlet_ id

shutter_ state

back ground

source_ id

estimated_ source_in _ shutter_x

estimated_ source_in_ shutter_y

Nov. 6, 2013


Source Info Extension

Table listing all sources Fields would include:

May include additional information from nirspec_sources_flux

Source id <ppppp>_<nnn> Where nnn is source number within program

Source name Source names standardized Format: <ppppp>_<nnn>_<ra>_<dec> Source names should be assigned within PPS Need to propagate source name from level 2 extensions to level 3 data

files

RA and Dec ICRS reference frame and J2000.0 epoch

- This information to go in the source extension header keywords.Nov. 6, 2013

source_id source name alias catalog_id ra dec

5-27

Imaging Association Introduction


5-28

DMS SDR3 Association Presentations

What is an association (Swade)

How to form an association pool (Swam/Alexov)

How to generate associations (Bushouse)

Separate association design review in 2014 with more details


5-29

6.3.1 Association Properties


5-30

6.3.4 Associations in the Archive


5-31

Goals of Design

Identify the associated (level 3) data products for each JWST science instrument Develop a naming convention for the associated data products

Identify individual files that will be associated with the data products

Identify the parameters necessary to generate each association

Identify the source of the parameter values


5-32

Association Design Process

Started with designing associations for imaging data products Imaging associations due in DMS Build 3

Considered SODRM programs as use cases Built MIRI and NIRCam imaging association pool and considered

data products for SODRM 93060 - Galactic Stellar Clusters


5-33

Association Components


5-34

Associated Data Products

Imaging Mosaic combined image NIRCam mosaic contains multiple detectors Dither combined image (if no mosaic)

Spectra NIRSpec Extracted 1-d and 2-d spectra over both detectors NIRSpec MSA Source Catalog (level 4) MIRI Extracted 1-d and 2-d spectra MIRI Extracted 1-d and 2-d spectra over both IFU detectors


5-35

Associated Files

Individual Exposures (single detector; levels 1a, 1b, 2a, 2b)

Jitter data

NIRCam Coronagraphic target acquisition

NIRSpec Target acquisition images

NIRSpec Confirmation images

NIRSpec MSA Pre-image

NIRSpec MSA Metadata file

NIRSpec Reference star file

NIRSpec Background exposures (may be shared among exposures)

MIRI Coronagraphic target acquisition images

MIRI Spectroscopic target acquisition images


5-36

Contemporaneous Reference Files

NIRCam flats

NIRSpec Auto-flats

NIRSpec Wave-cals

MIRI Imager internal flats

MIRI MRS internal flats

MIRI Chopped background image MIRI Different visit, no interrupts

Persistence darks (in the distant future)


5-37

Imaging Associations


5-38

Template Definition

Template – APT input used to define individual Observations. An Observation Template is completed in APT which provides the

necessary information for the internal scheduling software to generate the observing sequence (integrations, exposures, and visits) to execute on the telescope.

The templates are selected based on an observing strategy (e.g. MIRI imaging, MIRI low resolution spectroscopy) required for observations.

Only one instrument template can be used for any Observation.

An Observation is a single instance of a template. One-one correspondence exists between an observation and

template instance


5-39

Observation Data Products

In general, level 3 data products do not extend beyond one observation.

A mechanism is needed for PSS to let DMS know about exceptions: Multi-observation mosaics Shared contemporaneous calibration exposures Latent image correction (Defer until the distant future)


5-40

Imaging Data Products

Generate most combined/least combined data products Least combined – exposures processed through level 2b Most combined: Mosaic, Dither, etc.

Generate level 3 data products directly from all relevant level 2b exposures Not from any intermediate data products

Separate observation products generated for Each SI configuration (e.g., filter) NIRCam LW and SW channels


5-41

Dither Data Products

Dither data product generated if no mosaic May extend over multiple visits.

No intermediate data products are generated from sub-pixel dithers.

All exposures of the same target using the same filter combined into a single dither data product. All four NIRCam SW detector exposures combined into dither data

product. LW detector exposures combined into separate data product.


5-42

NIRCam Multi-Visit Dither


5-43

MIRI Imaging Dither Patterns

• The distribution of sampling positions for the Cycling, 12-point Reuleaux, and 5-point Gaussian patterns, shown at medium scale.

• The origin is the aperture center, where the target is specified.

• Sub-pixel sampling may also be included.• Dither pattern size depends on filter.


5-44

Mosaic Data Products

Mosaic data products combine all possible pointings in an observation. A mosaic is typically completely defined within an observation.

Except mosaic can be split into two observations if Not all tiles have guide stars Multiple position angles improve observing efficiency The first observation fails

No intermediate data products generated (e.g., dither data products).

Do not extend mosaics over programs in the pipeline. Standard pipeline does not address higher level data products.


5-45

Mosaic Observation

Courtesy of Jeff Valenti


5-46

Higher Level Data Products

Pipeline Association Baseline: Combine mosaics/dithers for a single observation (Level 3 data

product)

Higher (> Level 3) data products build by associating data outside routine science data processing pipeline Mechanisms and tools to identify exposures to associate and

facilitate combining exposures will help Example: Multiple observations from a single target in a program


5-47

Future Data Product Design Work


5-48

Future Data Product Design Work

Continue to develop association mechanisms and level 3 imaging data products (DMS Build 3) Association design review in 2014

Add keywords derived from observatory ephemeris to level 1b headers (DMS Build 3) Compute differential velocity aberration correction

Develop FGS guide star image/jitter pipeline and data products (DMS Build 4)

Add keywords from DMS engineering database to level 1b headers (DMS Build 4)

Develop level 2 spectral data products (DMS Build 5) Data array scheme for spectral data

Spectral association mechanisms and level 3 data products (DMS Build 6)


5-1 DMS Data Products Nov. 6, 2013 S&OC Data Management System Design Review 3.

Documents

Transcript of 5-1 DMS Data Products Nov. 6, 2013 S&OC Data Management System Design Review 3.