PACS IIDR 01/02 Mar 2001 On-Board Data Compression1 On-Board Data Compression Concept A. N....

On-Board Data Compression 1

PACS IIDR 01/02 Mar 2001

On-Board Data Compression Concept

A. N. BelbachirVienna University of Technology



Signal Description

• Photometry mode (Bolometers):– 16 bits detector signal– 40 Hz readout rate– SNR 15000 - 25000– data rate of 1600 Kbits/s– SWL detectors with 16x16x8 data values (1280

Kbits/s)– LWL detectors with 16x16x2 data values (320

Kbits/s)



Signal Description

• Spectroscopy mode (Photoconductors):– 16 bits detector signal– 256 Hz maximum readout rate– Reset interval from1/32s to 16s– SNR 450 - 650– data rate of 3600 Kbits/s– SWL detectors with 18x25 data values (1800 Kbits/s)– LWL detectors with 18x25 data values (1800 Kbits/s)– 18x26 data is the science input to the SPU.– Data rate is 3744 Kbits/s



Maximum Data Rate

The Maximum data rate = (science data + DEC/MEC header)/s

• Photometry: [(161610) channels 16bits + (260 8) header]

40 Hz = 1637.5 Kbits/s

• Spectroscopy: [(18262) channels 16bits + (2728) header]

256 Hz = 4032 Kbits/s



Downlink Rate and Compression Ratio

• The downlink rate is 100 Kbits/s• Compression ratio for raw data is at least

– 16.38 in photometry– 40.32 in spectroscopy

• The DEC/MEC header contains – The observation configuration

• Timing parameters• Readouts specifications• Observation identification...

– The compression parameters



Typical Signals

Simulation

PhotometryV

olt

ag

e

Time



Typical Signals

Simulation

Spectroscop

yV

olt

ag

e

Time



Typical Signals

SimulationSpectroscopy

Volt

ag

e

Time



SPU Data Flow

DPU

SPU SWL

DEC/MEC

SPU PSU

SPU LWL

S/C Power via DPU

28V Primary Voltage

Data

command/response / HK

compressed dataData

command/response / HK

compressed data



SPU Transmission Modes

• Compressed data mode: – Transmission of compressed data + header and raw

data of few channels.– Transmission of SPU HK

• Raw data mode: – Transmission of lossless compressed data from

selected detectors– Transmission of compressed header and SPU HK



SPU Transmission Modes

• Compressed data plus raw data mode– Switch between compressed data and raw data

transmission mode for the processing of the same data

– Transmission of compressed header and SPU HK

• Transparent mode– Transmission of data from selected detectors without

compression– Transmission of compressed header and SPU HK



Compression Parameters

• Default compression ratio• Double the compression ratio• Triple the compression ratio• Quadruple the compression ratio• Do not compress



DPU Commands

• Stop compression• Start compression• Perform SPU test in photometry• Perform SPU test in spectroscopy• Perform warm Reset• Start the program for the bolometer test mode• Load to SPU RAM (5 specific load commands)• Dump from SPU RAM• Start Peak-up Software



SPU HLSW Switching Modes

• 28 software modes in total– 18 compression modes according to DEC/MEC raw

data– 9 software modes according to DPU commands– SPU standby



SPU HLSW Flowchart

Yes

TOP

Data Input from DEC/MEC and/or

DPU

No

Communication I/F

Peak-up I/F

Warm Reset

Dump

YesNo

Stop OBS

Compression Mode Identification

Data Output from SPU to DPU

Communication I/F

Watch Process

Application SW Command

Indentification

Data Compression (+ OBS HK)

Send HK

DPU Command

Test Spec.

Load Start OBS

Test Phot.

Bol. Test



Reduction / Compression Scheme

100Kbit/s

Spec

tros

cop

y 1800 Kbit/s

1800 Kbit/s

Ph

otom

etry 320 Kbit/s

1280 Kbit/s

Integ-ration

Robust Avera-ging

Glitch Detec-

tion

Sample Rejec-

tion

Temporal Redun. Reduc.

Spatial Redun. Reduc.

Raw Data

Selection

Lossl. Coding

Ramp Fit-ting

Glitch Detec-

tion

Integ-ration

Ramp Rejec-

Temp. Red.

Reduc.

Spat. Red.

Reduc.

Raw Data Selection

Lossl. Coding

Pre-pro-ces-sing



Preprocessing

• This module is still TBD

• Ramp linearization necessary or not



Robust Averaging and Ramp Fitting

• Robust averaging in photometry– Preprocessing (TBD)

– Median fitting

– Mean value calculation

• Ramp fitting in spectroscopy– Fitting with RANSAC

– Least square fitting



Glitch Detection

• To ensure not to integrate over invalid sensor readings (i.e. glitches)

• Multilevel glitch detection– Intrinsic deglitching at individual sample level – Extrinsic deglitching at ramp/averages level and

considering subsequent slopes/averages



Integration in Spectroscopy

• On-board integration of sensor readings to achieve the desired compression ratio

• Special emphasis:– Guarantee integration over the right readings (i.e.

synchronized with positions of chopper)– Not to integrate over ramps affected by glitches

• 2 step process:1. Discard all data of CRE integration blocks with glitches2. Add slope data of successive ramps within the same

chopper position



Redundancy Reduction

• Temporal redundancy reduction– Calculation of references frames and differences

• Spatial redundancy reduction – Calculation of references pixels and differences



Lossless Coding

• Redundancy reduction reduces magnitude of pixel’s values

• Implement the run-length encoding algorithm to achieve additional compression



Compression Ratio from Redundancy Reduction (Worst Case)

• Photometry: – SNR: 15000 for 16 bits signal– 4 bits for the noise and 12 bits for the offset signal– for 12 averages: the compression ratio is 3.2

• Spectroscopy:– SNR: 450 for 16 bits signal– 10 bits for the noise and 6 bits for the offset signal– for 2 slopes: the compression ratio is 1.23



Compression Ratio from Lossy Compression (Worst Case)

• Photometry: – 5 samples to average in order to fulfill the telemetry

requirements of 100 Kbits/s– Total compression ratio: 3.2 5 = 16

• Spectroscopy in the most challenging case with a reset interval of 1/32s:– Compression ratio of 8 from the ramp fitting– Integration of 4 ramps– Total compression ratio: 8 4 1.23 = 39.38

• The remaining ratio is achieved by the spatial redundancy reduction and the lossless coding



Compressed data rate

• Photometry: Compressed science data + compressed header + no

data from additional channels 100.084 Kbits/s without the spatial redundancy

reduction and lossless coding algorithms

• Spectroscopy: Compressed science data + compressed header + raw

data of 2 channels per board 99.983 Kbits/s for the most challenging case



Data Stream from DEC/MEC in Photometry (SWL SPU)

array 6 array 7array 5

17

32

array 2 array 3array 1

1

16

1 16 17 32 33 48

array 4

array 8

49 64row

column

header1 256

array 1

1 256

array 2

1 256

array 3

header1 256

array 5

1 256

array 6

1 256

array 7

header1 256

array 4

1 256

array 8



Data Stream from DEC/MEC in Photometry

Data Stream:

Multiple of 4.2K for the SWL SPU

Block1

Frame 1Chopper Move (max. 10 Hz)

~4.2K

Next Frames

Block2~1.56

K~1.56K

Block3~1.06K

Multiple of 1.06K for the LWL SPU



Data Stream from DEC/MEC in Spectroscopy

Data Stream:

Multiple of 1K per each SPU board

Frame1

Block1Chopper Move(max. 10 Hz)

~2K

Next Frames

Frame2

~1K ~1KFrame3~1K

Block2

~2K

Frame4~1K



Memory Constraints

Buffer (64KB)

Processing Memory

(1.44MB)

To/From DPU 100 Kbits/s

From DEC/MEC 4032Kbits/sBuffer

(0.5MB)

Buffer (0.5MB)

Buffer switch of Data Input

Program Memory (1.5 MB)

Link Buffer (32KB)

To/From SPU

Board

SPU RAM Organization



Memory Organization Requirements

• Start Processing with 0.5 MB buffer– 2s of observation in Spectroscopy (36KB for the

header)– 3s of observation with blue detectors in photometry

(21KB for the header)– 12s of observation with red detectors in photometry

(28KB for the header)

• 32 bits address in the data memory• 16 bits signal per detector data• 2 readouts data per 32 bit address



Memory Requirements

• 1.5 Mbytes of RAM for the program memory

• 2.5 Mbytes of RAM for the data memory– 2 buffers of 0.5 Mbytes to store DEC/MEC

raw data– 1 buffer of 64 Kbytes to store the

compressed data + HK– About 1.4 Mbytes for data processing



Processing Requirements in Spectroscopy (Worst Case)

• n = 8 samples at 256Hz (Nb. of samples per ramp)• R = 64 ramps for the most challenging case 1/32s reset• Processing Power:

– Ramp fitting computationally most demanding20 FLOPs for 1 Fit10 FLOPs for 1 Residual20 FLOPs per sample for glitch detection20 FLOPs per ramp for glitch detection20 FLOPs for the integration of 4 ramps20 FLOPs for the redundancy reduction



Processing Requirements in Spectroscopy

• For 450 detectors we have 4500 . R(n2 + 3n + 3)

• For 2 s observation 26MFLOPs/2s=13MFLOPs/s



Processing Requirements in Photometry (Worst Case)

• n = 5 samples at 40Hz (Nb. of samples to average)• A = 20 averages in a 3 s buffer for the SWL SPU• Processing Power:

– Robust averaging computationally most demanding40 FLOPs for 1 robust averaging10 FLOPs for 1 Residual20 FLOPs per sample for Glitch detection20 FLOPs per average for glitch detection20 FLOPs for the redundancy reduction



Processing Requirements in Photometry

• For 2048 detectors we have 2048 30 A(n + 4)

• For 3 s observation 11MFLOPs/3s = 4MFLOPs/s



Compressed Entity

• Blocks:Nb. of blocks the compressed entity packet has been split into (the block unit is 1004B)

• Type: Type of the observing Mode (spectroscopy or photometry)

• PIXB/PIXR: relationship between the HK and the science packets ‘the same index’

• CDMHS: Compressed DEC/MEC Header Size

• CDMH: Compressed DEC/MEC Header field

• DECID: Decompression Code ID

• CSD: Compressed Science Data

Compressed Entity Packet Structure

CSD (TBD Bytes)

PIXB/PIXR (2

Bytes)

Blocks (1Byte)

CDMH (CDMHS Bytes)

DECID (TBD Bytes)

Type

(1Byte)

CDMHS

(1 Byte)



Compressed Entity/HK

• The compressed entity size is: – ~ 31KB per SPU board in photometry

– ~ 13 KB per SPU board in spectroscopy

• The SPU HK: compression results/SPU status– SPU HK rate depends on the reset interval and

compression efficiency

– SPU HK size is 38 Bytes



Summary

• The concept of the on-board software finalized

• Compression of the science data according to the header

• DPU commands to load or dump from the SPU memory or to perform specific exercises

• HK to inform about the SPU status

• Two different set of compression modules according to:– Photometry to achieve at least a compression of 16

– Spectroscopy to achieve at least a compression of 40

PACS IIDR 01/02 Mar 2001 On-Board Data Compression1 On-Board Data Compression Concept A. N....

Documents

Transcript of PACS IIDR 01/02 Mar 2001 On-Board Data Compression1 On-Board Data Compression Concept A. N....