6e-1 Science Data Products Daryl Swade DMS Systems Engineer S&OC System Design Review #1.

6e-1

Science Data Products

Daryl Swade

DMS Systems Engineer

S&OC System Design Review #1

6e-2

Intro/Overview

Science Data Product design shows what the DMS software should produce. Software design then shows how to generate data products.

JWST science data products generated by DMS are dynamic due to updates to calibration reference data and algorithms, increased knowledge of the science instruments learned in the

early stages of the mission, and DMS software error corrections.

Intent of DMS Science Data Product Generation is to provide data users with basic reduced data products they need to begin detailed analysis.

DMS data products conform to Flexible Image Transport System (FITS) convention. standard archival format for astronomical data


6e-3

DMS Data Flow – Data Product Generation


6e-4

JWST Up-the-Ramp Readout


6e-5

Data Level Definition


Science Data Archive

Object Catalog

Level 0: Science telemetry packet files

Level 1a: Original FITS files

Level 1b: Uncalibrated FITS files

Level 2a: Ramp calibrated exposure FITS files

Level 2b: Fully calibrated exposure FITS files

Level 4: Products of data analysisExtractedobjects

Proposer

Level 3: Calibrated, combined FITS files

Level 0.5: Pod files with packets sorted by exposure/detector

JWST

6e-6

Science Data Product Inputs


DataManagementSubsystem

(DMS)

ProposalPlanning

Subsystem (PPS)

Flight Operations Subsystem

(FOS)

Project Reference Database

Subsystem (PRDS)

Recorded Science Data Files

Observatory Status Files

Calibrated Engineering Data Products

Science Instrument Aperture Files

Datamode Table

Detector Table

Leap Seconds Table

S&OC PRD Version Name

Proposal Information

Planning Information

NIRSpec MSA Pre-images

Observatory Ephemeris

6e-7

Generation of Original Exposure Data Files


science telemetry processing

Level 0 data file

Read science image data

Observatory Status file Data partitioning

o sort data into files by detector and exposure

Reformat data to FITS

Standard FITS keyword information

o generate standard FITS keywords

o include keyword information from science telemetry headers

Level 1a FITS file

o read science image data from science telemetry files

Level 0.5 pod file

6e-8

Level 1 FITS File Structure


Science Extension

Extension Header

Extension DataMIRI Reference Output Extension

Extension Header

Extension DataGroup Information Extension

Extension Header

Extension Data

Primary Header

6e-9

Science Data Cube

single exposure, single detector


6e-10

Image Orientation

Detector and science image frame orientations as seen looking through the optics toward the sky


red – detector frameblue – science framegreen – apertures

NIRCam

NIRSpec

MIRI

NIRISSFGS1 FGS2

Transformation requires rotation, parity flip, and/or shift.

6e-11

Information Flow to Level 1 FITS Headers


Recorded Science Data

File

Observatory Status File

PPS Database

Detector Table

Datamode Table

Science Instrument Aperture File

Level 1a FITS Header

Level 1b FITS Header

Observatory Ephemeris

FOS PPS PRDS DMS

Engineering Database

Calibrated Engineering Data

6e-12

Level 1a-Level 1b Header Keyword Population


engineering

time

o wait for engineering data from the FOS

o incorporate engineering data into science data headers

Incorporate engineering data

Perform Barycentric time

correction

o convert UTC to barycentric time for a reference pixel of each SCA

Perform Heliocentric time

correctiono convert UTC to heliocentric time

for a reference pixel of each SCA

proposal, planning, and scheduling

Populate PPS infoo populate keyword values

with proposal, planning, and scheduling information

Verify telemetry

o compare information from the science telemetry packet header and engineering parameters to PPS database entries

PPS replicated database tables

position

o populate header keywords with JWST position and velocity information

Determine s/c position and

velocity

Calculate WCS parameters

o translate focal plane coordinates to sky coordinates

o determine differential velocity aberration correction

Level 1b FITS file

Calibrated engineering data

Level 1a FITS file

s/c ephemeris

data

Science Instrument

Aperture File

Observatory Status file Populate OSF info

o populate keyword values with Observatory Status File information

6e-13

Files in an Exposure Data Set

File type Data level created

Science Instrument

Format

uncalibrated 1b all FITS pixel data image and group information ASCII table

MSA metadata 1b NIRSpec FITS ASCII table

reference stars 1b NIRSpec FITS ASCII table

jitter file 1b all FITS ASCII table

count rate image 2a all FITS image

calibrated image 2b all FITS image

processing history 1a-2b all text


6e-14

Exposure File Naming Convention

At data levels 1 and 2, the science data files are FITS format files that contain the pixel values

for a single exposure from a single detector.

Goals for file name: DMS data user able to map the data file to the observation input through APT. Operations staff can map every exposure to a construct in the JWST commanding scripts.

The file name is based on the following exposure hierarchy: Program (PPPPP) Observation number (OOO) Visit (VVV) Visit Group (GG) Parallel Sequence id (S, 1 prime, 2-5 parallel) Activity (AA, base 36) Exposure number (EEEEE)

Example: jw<PPPPP><OOO><VVV>_<GGSAA>_<EEEEE>_<detector >_<suffix> .fits jw12345012009_01101_00012_nrcalong_uncal.fits

Visit ID - <PPPPP><OOO><VVV> - explicitly embedded in exposure file name

Exposure number resets with each new activity The exposure number within the file name does not necessarily monotonically increase for each

exposure within a single visit.S&OC System Design Review #1

6e-15

Guide Star Pipeline

Guide Star Pipeline detailed design to be addressed in DMS SDR4

Jitter data file generated by Guide Star Pipeline Separate process from science data pipelines Data product created at visit level

Guide Star (GS) acquisition images About GS - Full frame image, Acq 1: 128x128, Acq 2: 32x32 At science attitude: Acq 1: 128x128, Acq 2: 32x32

Fine guide box: 8x8 @ 16Hz (64 msec.) May not want to save all 8x8 fine guide data Centroids in engineering data for jitter

Include additional pointing information in engineering data


6e-16

NIRSpec MSA Data Flow


Data Processing

Proposal Input

MSA Pre-

image

MSA Planning

Tool

PPS Database

APT

Science Data

Processing

Recorded Science Data File

Level 1b Science Data File

Level 2 Calibration

Level 2b Science Data File

Level 3 Calibration

Level 3 Science Data File

MSA Metadata

File

JWSTDetector/

exposure based

Source based

PI

Source Catalog

Data product key:Ancillary dataExposure dataCombined data

6e-17

NIRSpec MSA Metadata File Structure


Data array for each shutter planned open or closed

Slitlet information, background shutter, fraction of source in shutter

Source id, source name, alias, RA, Dec

6e-18

Generation of Calibrated Exposure Data Files


Level 1b FITS file

calibration reference files

calibration

Perform up-the-ramp calibration

o remove detector specific signatures

Determine calibration

reference files

o determine appropriate calibration reference files and switches

Perform instrument specific calibration

Calibration reference

files

Level 2a data file

Level 2b FITS file

CRDS

6e-19

Level 2 FITS File Format


Primary Header

Data Quality Extension

Extension Header

Extension Data

Science Extension

Extension Header

Extension Data

Error Extension

Extension Header

Extension Data

Additional Extension(s)

Extension Header

Extension Data

MIRI Reference Output Extension

Extension Header

Extension Data

6e-20

Level 3 Data Products - Associations

Relationships between multiple exposures are captured in an association.

An association is a means of identifying a set of exposures that belong together and may be dependent upon one another.

The association concept permits exposures to be calibrated, archived, retrieved, and reprocessed as a set rather than as individual objects.

For initial processing, an association and its level 3 data products will not extend beyond a single observation. Design will include the capability to support multi-observation and

multi-program associations.

For each association, DMS will generate the most combined and least combined data products. Intermediate data products will not be saved. For imaging associations, the most combined will be either a

mosaic or a dither product.


6e-21

Association Components


Science Instrument

Data Products Related Files Contemporaneous Calibration Files

NIRCam • Mosaic combined image (multiple detectors)

• Dither combined image (if no mosaic)

• Individual exposures (single detector; levels 1a, 1b, 2a, 2b)

• Coronagraphic target acquisition

• Jitter data

• Flats (taken with science exposure)

• Persistence darks

NIRSpec • Extracted 1-d and 2-d spectra over both detectors

• MSA Source Catalog (level 4)


• Target acquisition images

• Confirmation images

• Jitter data

• MSA Pre-image

• MSA Metadata file

• Reference star file

• Background exposures (may be shared among exposures)

• Auto-flats

• Wave-cals

MIRI • Mosaic combined image


• Extracted 1-d and 2-d spectra

• Extracted 1-d and 2-d spectra over both IFU detectors


• Coronagraphic target acquisition images

• Spectroscopic target acquisition images

• Jitter data

• Imager internal flats

• MRS internal flats

• Chopped background image (different visit, no interrupts)


NIRISS/FGS • Mosaic combined image



• Jitter data

• Point source image for PSF characterization


6e-22

Association Data Products

Association Pools Created by Science Data Processing Contains a list of all potential exposures that may form an

association

Association Table Created in calibration pipeline environment from Association Pool

information Contains list of exposures to be used to create a single Level 3

Data Product

Level 3 Data Products Formed from exposures listed in Association Table


6e-23

Association Processing

All exposures – even single, unassociated exposures – will have a level 3 data product. Well-established image rectification software (Drizzle) will be run at

level 3 on all exposures. All data will get a source based, level 3 file name.


Assemble exposures

from level-0 data

Level-1 processing

Association pool

Uncaibrated exposure

Level-2a processing

Partially calibrated exposure

Level-2b &Level-3

processing / Association Generator

Associated data

products

Hold for all observation exposures

6e-24

Primary Header

Science Extension

Extension Header

Extension DataHeader Table Extension

Extension Header

Extension Data Additional Extension(s)

Extension Header

Extension Data

Level 3 Image FITS File Format


Most combined image

FITS table of selected keywords from input exposures

Additional extensions to be defined by calibration working group

6e-25

Level 3 Data Product File Name

Shift in naming convention for level 3 data product file names

The association data product name will provide an indication of the file content. Based on data discovery lessons learned from MAST High Level Science Product

(HLSP) and Hubble Legacy Archive (HLA) experience, A proposed name for the JWST level 3 data products was patterned after the

HLSP data file names. (http://archive.stsci.edu/hlsp/hlsp_guidelines.html#Filenames).

The format for the level 3 data product file name would be:

jw<ppppp>_<target(epoch)>_<science_instrument>_

<optical_element>_ <product_name>.fits <ppppp> is the program identifier <target> is the target name, which may contain proposer defined non-

alphanumeric characters and an epoch <science_instrument> is nircam, nirspec, miri, niriss, or fgs < optical_element > is the optical element or optical element combination used <product_name> indicates the type of data product, e.g., image

Examples from SODRM: jw94010_m31_nircam_f356w_image.fits jw95020_cdfs-07-e1_nircam_f115w_image.fits jw93060_w51a+g49.5-0.4_miri_f1500w_image.fits


http://archive.stsci.edu/hlsp/hlsp_guidelines.html#Filenames

http://archive.stsci.edu/hlsp/hlsp_guidelines.html#Filenames

6e-26

Data Level Format Summary

Data level Format

Level 0 CCSDS packets

Level 0.5 CCSDS packets sorted by exposure and detector

Level 1a 4-d data cube

Level 1b 4-d data cube

Level 2a 3-d data cube – collapse groups within each integration into a count-rate imageandSingle 2-d image – collapse integrations into one count-rate image

Level 2b Single 2-d image

Level 3 Single or combined, rectified 2-d image; extracted spectra


6e-1 Science Data Products Daryl Swade DMS Systems Engineer S&OC System Design Review #1.

Documents

Transcript of 6e-1 Science Data Products Daryl Swade DMS Systems Engineer S&OC System Design Review #1.