Post on 14-Jan-2016
SPACETELESCOPESCIENCEINSTITUTE
Operated for NASA by AURA
COS Pipeline PDR
File Structure Overview
Charles D. (Tony) Keyes
7 December 2000
COS PIPELINE PDRTony Keyes
December 7, 2000 COS File Structure Overview
Space Telescope Science Institute
2 of 13
COS Data Format 2-d data: FITS files containing imsets consisting of science and 2-d data: FITS files containing imsets consisting of science and
data quality images data quality images (unlike STIS, no error array)(unlike STIS, no error array) 1-d data and TIME-TAG lists: FITS tables containing fluxes, 1-d data and TIME-TAG lists: FITS tables containing fluxes,
wavelengths, errors, quality, positions, and other related wavelengths, errors, quality, positions, and other related quantitiesquantities
May explore compression for reduced dataset volumeMay explore compression for reduced dataset volume Data from each FUV detector segment stored as separate imset Data from each FUV detector segment stored as separate imset
series within a single fileseries within a single file– All associated data (e.g., wavecals, FP-SPLIT components, REPEATOBS All associated data (e.g., wavecals, FP-SPLIT components, REPEATOBS
components, STIMs) use separate files components, STIMs) use separate files (unlike STIS)(unlike STIS)
COS PIPELINE PDRTony Keyes
December 7, 2000 COS File Structure Overview
Space Telescope Science Institute
3 of 13
Data File Naming Conventions Product Suffix Type Contents --------------------------------------------------------------- Uncalibrated _raw image Raw counts image _rtg table Raw TIME-TAG event list _asn table Association file
_trl table Trailer file (input)
_lrc image Local rate check image [TBD]
_lsp text LRC support file (header) [TBD] _jit table _jif image _pha table Pulse-height (FUV ACCUM only) _rs1 image Stim-pulse 1 (FUV ACCUM only) _rs2 image Stim-pulse 2 (FUV ACCUM only)
COS PIPELINE PDRTony Keyes
December 7, 2000 COS File Structure Overview
Space Telescope Science Institute
4 of 13
Data File Naming Conventions (continued) Product Suffix Type Contents ------------------------------------------------------------------------------------------------------------------------------ Calibrated _ctg table Calibrated TIME-TAG event list
_flt image Flat-fielded, corrected counts _fls image Summed flat-fielded corrected-counts _x1d table 1-D extracted spectra _x1s table Summed (combined) 1-D extracted spectra)
_cs1 image Corrected stim-pulse 1 (FUV ACCUM only) _cs2 image Corrected stim-pulse 2 (FUV ACCUM only) _trl table Trailer file (output) _spt image Support file (header) ------------------------------------------------------------------------------------------------------------------------------ Wavecal exposures produce same calibrated files as science, except there Wavecal exposures produce same calibrated files as science, except there
may be no flux calibration applied to the _x1* files (TBD). Dark and flat may be no flux calibration applied to the _x1* files (TBD). Dark and flat exposures produce no _x1* files. Acquisition exposures produce no calibrated exposures produce no _x1* files. Acquisition exposures produce no calibrated products. products.
COS PIPELINE PDRTony Keyes
December 7, 2000 COS File Structure Overview
Space Telescope Science Institute
5 of 13
COS Observational Quantity Definitions x = raw detector x-position (I*2) y = raw detector y-position (I*2) t = time of photon detection (R*4) ph = pulse-height (I*1) xcorr = corrected detector x-position (R*4) ycorr = corrected detector y-position (R*4) EPS = photon list sensitivity-parameter (R*4) obsC = observed counts (R*4) C = corrected counts (R*4) q = data quality (I*1) lambda = wavelength (R*4) F = calibrated flux (F-lambda) (R*4) G = gross extracted counts (R*4) N = net extracted counts (R*4) B = background extracted counts (R*4) S = sensitivity function (R*4)
COS PIPELINE PDRTony Keyes
December 7, 2000 COS File Structure Overview
Space Telescope Science Institute
6 of 13
File Contents: Observational Quantities _rtg: contains x,y,t,ph (no ph for NUV)
– FUV: 9 bytes/event; NUV: 8 bytes/event _ctg: contains xcorr,ycorr,t,ph,EPS,q,lambda (no ph for NUV)
– FUV: 22 bytes/event; NUV: 21 bytes/event _raw: contains obsC
– FUV: 32 Mbyte/file; NUV: 4 Mbyte/file _flt: contains C,q (in each imset)
– TIME-TAG: FUV: 160 Mbyte/file; NUV: 5 Mbyte/file– ACCUM: FUV: 40 Mbyte/file; NUV: 5 Mbyte/file
_fls: same as _flt “_obs”: contains obsC (in each imset) [size same as _raw] _x1d: contains F,lambda,G,N,B,S,q (in each imset)
– FUV: ~1 Mbyte/file; NUV: ~0.08 Mbyte/file _x1s: same as _x1d _rs1: contains obsC (same for _rs2) _cs1: contains C (same for _cs2)
COS PIPELINE PDRTony Keyes
December 7, 2000 COS File Structure Overview
Space Telescope Science Institute
7 of 13
Exposure type rt
g ra
w
ctg
pha
“obs
” fl
t fl
s x1
d x1
s rs
1 rs
2 as
n tr
l sp
t
TIME-TAG No FP FUV NUV
X X
X X
X X
X X
X X
X X
X X
X X
TIME-TAG FP FUV NUV
X X
X X
X X
X X
X X
X X
X X
X X
ACCUM No FP FUV NUV
X X
X X X
X X
X X
X X
X X
X X
X X
X X
ACCUM rptobs FUV NUV
X X
X X X
X X
X X
X X
X X
X X
X X
X X
X X
X X
ACCUM FP FUV NUV
X X
X X X
X X
X X
X X
X X
X X
X X
X X
X X
X X
IMAGE T-TAG X X X X X X
IMAGE ACCUM X X X X X
IMAGE ACCUM rpt X X X X X X
ACQ and PEAKUPs X WAVECAL X X X X X
FLAT and DARK X X X X
COS PIPELINE PDRTony Keyes
December 7, 2000 COS File Structure Overview
Space Telescope Science Institute
8 of 13
Time-Tag Data File Structure
ext 1 - b inary tab le
ext 1 - b inary tab le
ext 2 - b inary tab le detector B
Time-tag /FPsplit = no
NUV FUV
ext 1 - b inary tab le
PHUext 1 - b inary tab le detector A
PHU
ext 1 - b inary tab le
PHU
Time-tag / FPsplit = yes
Lpppssoot_RTG.FITS
PHU
PHU
ext 1 - b inary tab le
PHUexposure 4 oo = 04
exposure 3 oo = 03
exposure 2 oo = 02
exposure 1 oo = 01
ext 2 - b inary tab le detector Bext 1 - b inary tab le detector A
PHU
ext 2 - b inary tab le detector Bext 1 - b inary tab le detector A
PHU
ext 2 - b inary tab le detector Bext 1 - b inary tab le detector A
PHU
ext 2 - b inary tab le detector Bext 1 - b inary tab le detector A
PHU
COS PIPELINE PDRTony Keyes
December 7, 2000 COS File Structure Overview
Space Telescope Science Institute
9 of 13
ACCUM / FPsplit = no Data File Structure
ext 4 - detector Bdata quality array
ext 3 - detector Bscience data array
im set 2
ext 2 - data qualityarrayACCUM /
FPsplit = no
NUV FUV
ext 1 - sciencedata array
PHU
Lpppssoot_RAW .FITSLpppssoot_RS1.FITSLpppssoot_RS2.FITS
im set 1
ext 2 - detector Adata quality array
ext 1 - detector Ascience data array
PHU
im set 1
COS PIPELINE PDRTony Keyes
December 7, 2000 COS File Structure Overview
Space Telescope Science Institute
10 of 13
ACCUM / FPsplit = yes or REPEATOBS=NData File Structure
NUV FUV
Lpppssoot_RAW .FITSLpppssoot_RS1.FITSLpppssoot_RS2.FITS
ACCUM /FPsplit = yes ext 2 - data quality array
ext 1 - science data arrayPHU
ext 4 - detector B DQ arrayext 3 - detector B sci array im set 2
ext 2 - detector A DQ arrayext 1 - detector A sci array
PHUim set 1
exposure 4 oo = 04
exposure 3 oo = 03
exposure 2 oo = 02
exposure 1 oo = 01
ext 2 - data quality array
ext 1 - science data arrayPHU
ext 2 - data quality array
ext 1 - science data arrayPHU
ext 2 - data quality array
ext 1 - science data arrayPHU
ext 4 - detector B DQ arrayext 3 - detector B sci array im set 2
ext 2 - detector A DQ arrayext 1 - detector A sci array
PHUim set 1
ext 4 - detector B DQ arrayext 3 - detector B sci array im set 2
ext 2 - detector A DQ arrayext 1 - detector A sci array
PHUim set 1
ext 4 - detector B DQ arrayext 3 - detector B sci array im set 2
ext 2 - detector A DQ arrayext 1 - detector A sci array
PHUim set 1
COS PIPELINE PDRTony Keyes
December 7, 2000 COS File Structure Overview
Space Telescope Science Institute
11 of 13
File Naming TIME-TAG “FP-SPLIT=NO" exposure:
Wavecal exposure science exposure (401s) (402s) association file: 4010_asn
wavecal exposure files: 401s_rtg 401s_ctg 401s_flt 401s_x1d
science exposure files: 402s_rtg 402s_ctg 402s_flt
association science product file: 4010_x1d
COS PIPELINE PDRTony Keyes
December 7, 2000 COS File Structure Overview
Space Telescope Science Institute
12 of 13
File Naming TIME-TAG FP-SPLIT=YES exposure:
Wavecal exposure science exposure (401s) (402s) science exposure science exposure (403s) (404s) science exposure (405s) association file: 4010_asn
wavecal exposure files: 401s_rtg401s_ctg
401s_flt 401s_x1d
science exposure 1 files: 402s_rtg402s_ctg
402s_flt 402s_x1d
science exposure 2 files: 403s_rtg403s_ctg403s_flt
403s_x1d science exposure 3 files: 404s_rtg
404s_ctg404s_flt 404s_x1d
science exposure 4 files: 405s_rtg405s_ctg405s_flt 405s_x1d
association science product file: 4010_x1d
COS PIPELINE PDRTony Keyes
December 7, 2000 COS File Structure Overview
Space Telescope Science Institute
13 of 13
COS Typical File Sizesexposure POD
(Mb) rtg raw
(Mb)
ctg (Mb)
flt (Mb)
“counts” (Mb)
OPUS (Mb)
into HDA (Mb)
TIME-TAG 18 45 110 160 160 493 63
ACCUM FUV
16 32 --- 40 40 128 48
ACCUM NUV
2 4 --- 5 5 16 6
WAVECAL
<18 <45 <110 160 160 320-376
<63
for 3000 sec FUV exposure: TIME-TAG produces 18 Mbyte full buffer of data for F-lambda=3e-14; TIME-TAG=ACCUM OPUS file size crossover at 1.e-14: for fainter TIME-TAG OPUS fileset smaller; onboard memory usage equivalent at 2.8 e-14; 6 Mb of TIME-TAG photons produces OPUS volume = ACCUM std volume