Digital Packaging Processor

Gordon Hurford

Jim McTiernan

EOVSA PDR 15-March-2012

Role of Digital Packaging Processor

• To filter, average, partially calibrate and convert raw correlator output into a Miriad-compatible format that is written to Interim Data Base

• Real time, irreversible processing

1

1

DPP Interface Overview

DPP

State Frame

ACC

Correlator

Start / End Scan Commands

Scan-independent Calibration Parameters

Scan Parameters

Frame parameters

Frame status report

<P>, <P2>,

Correlations

Interim Data Base

Miriad

format

Internal RFI

Database1 s timing tick

0.02 s timing tick

RFI

results

DPP Task Timescales• Data frame (20 ms)

– filter, and frequency-average correlator output

• Spectral frame (1 s) – Assemble, pre-calibrate, reformat and write data to Interim database

• Scan initiation– Accept, store and preprocess scan-specific parameters

• Occasional – non operational– Accept, store and preprocess calibration parameters

• TBD – Format results and write to RFI database

DPP Software Architecture

ACC State Frame Correlator IDB

DPP_CONTROL

I/O, data assembly, Miriad formatting, no numerical processing

DPP_Process_Header

DPP_Process_Dataframe

DPP_Process _Spectral _Frame

Communication via Common Blocks

Conventional, time-independent processing tasks with “Clean” inputs

C1

C2 C3, C4 …. C2

Cn = core within a mullti-core processor or nodes in a cluster

DPP

RFI database

DPP_Process_Parameters

C2

DPP_CONTROL-- Jim McTiernan --

• Handles all external I/O• Passes reformated header and calibration data

on to processing tasks• Assembles data frame input from correlator and

calls processing task only if/when complete• Assembles processed 20ms interval data and

calls spectral frame processor only when all accumulations are available

• Converts output of spectral frame processor to Miriad format

DPP_PROCESS_DATAFRAME• Tasks

– Identify RFI-affected subbands as a function of frequency only.– Combine with pre-flagged subbands to generate a “destination

vector” for each subband– Save RFI statistics – details TBD– Average subband data into spectral channels

• Challenge: 450 MB/s – Each visibility is handled only once– Multiple processors (up to 6) handle successive dataframes

• Initial Restriction:– No calibration at the subband level

DPP_PROCESS_SPECTRAL_FRAMEEvery spectral frame (1s)

• Convert antenna-based flags (e.g. slewing) from state frame to baseline-based, frequency-independent flags

• Apply time-independent complex gains • Correct for attenuator settings• Fine delay corrections-------------------------------------------------------• Baseline corrections

– (not necessary)• Apply non-linearity corrections

– Refinement to be added later, if necessary• Correction for spectral simultaneity

– not needed if using Miriad wide-band mode• Conversion of visibility, uv and analysis-relevant state-frame

data to Miriad-compatible format– May be a included here, in DPP_Control or as a separate Process

DPP-Correlator Interface• 4096 x 2 x16^2 8-byte visibilities

• 4096 x 2 x 16 8-byte <P2> values 9 MB / data frame 450 MB/s

• Two Dedicated 10 Gb Ethernet link(s)

• Interface architecture driven by correlator design

Format of Correlator Output

• Format of interface based on 3-Feb-2012 EST memo from Nimish, as clarified in 12-March-2012 email

• Features:– Accumulation divided into labeled packets, each with an

identical header (except for sequence numbers)– Packets divided into ‘chunks’, each of which contain all power,

power^2 and correlated data for a given subband– Data for a given subband has both polarizations interspersed,– All non-header data are scaled 4-byte integers

• Refinement:– Agreed that correlator output should always correspond to 16-

antenna/2 polarizations. – DPP will discard unused antennas/baselines

DPP-Correlator Interface Open Issues

• Specifics of EST format generalization to EOVSA

• Byte order: – Correlator outputs Big-endian (MSB first)– Many machines (e.g. PC’s) use Little-endian (LSB first)

• How is the load to be divided between the two Ethernet interfaces?– Current format (and DPP software) assumes sequential input within a

given accumulation

DPP Status• Overall:

– Software architecture and tasks defined– Platform to be acquired

• Interfaces:– IDB – Miriad format feasibility demonstrated with EST data– Correlator - as defined by Nimish with a few remaining issues– State frame - will follow protocol defined by GN, details TBD– ACC i – definition at an early stage– Timing ticks – TBD– RFI –TBD

• Milestones – Detailed software definition, coding and testing (with EST data) is underway– DPP_CONTROL has successfully associated tasks and processors– EST data loads correctly into DPP input common block– DPP_PROCESS_DATAFRAME well-defined and being coded.– EST to Miriad format conversion has been successfully demonstrated.– Immediate goal: End-to-end testing with EST data

Some DPP Open Issues• Do we use spectral or wideband mode in Miriad?

– Probably either would work – Initiall implementation will use wide-band mode

• (simpler, more flexible but less efficient) – analysis timing tests needed to see if this is an issue– easy to switch to spectral mode

• Is GNU Fortran 95 the best compiler choice?– Drawback is lack of support for STRUCTURE statements

• Workaround available for DPP_PROCESS_DATAFRAME

• Options for software development platforms – Independently develop DPP_CONTROL and processing routines on

‘personal’ machines, then test at OVRO?– Duplicate platform located in Bay area (JM, GH or SSL?)– Duplicate platform at NJIT ?

• Need to identify and purchase a multi-processor platform for DPP in near future.