SMOS SAG, November 2-3, 2006 SMOS Data Processing Ground Segment (DPGS)

SMOS SAG, November 2-3, 2006

SMOS

Data Processing Ground Segment (DPGS)


Function ESAC (Villafranca)

ESRANGE (Kiruna)

Payload planning, TC generation, and monitoring

SPGF,PLPC-PXMF -

X-band acquisition and Level 0 generation

XBAS -Processing infrastructure: inventory, storage, data distribution, monitoring and control

PDPC-Core MMFI

Processing and monitoring L1OP,L2OP,SPQC,

MF

Reprocessing Centre

Analysis Tools IATs -Archiving - MMFI

Product Ordering via User Services - MMFI

CEC

New

SMOS Ground Segment


SMOS Ground Segment

S-band X-bandAuxiliary Data

Providers

SOGS

PLPC

SPGF

XBAS

PXMF

Fast Processing Centre

Long Term Archive

CEC

IAT

MF

SPQC

Refer.Processor

Level 0

PDPC Core

Level 1

Level 2

Reprocessing Centre

Level 1

Level 2 MF

SPQC

Local Archive

Data Dissem.

Multi-mission Facility

Infrastructure

DPGS

Multi-mission User Services

USERS

ReportsData

Subscriptions

Data On-request


SMOS Ground Segment


Providers

SOGS

PLPC

SPGF

XBAS

PXMF


Long Term Archive

CEC

IAT

MF

SPQC

Refer.Processor

Level 0

PDPC Core

Level 1

Level 2

Reprocessing Centre

Level 1

Level 2 MF

SPQC

Local Archive

Data Dissem.


Infrastructure

DPGS


USERS

ReportsData

Subscriptions

Data On-request

Indra Espacio responsibility


SMOS Ground Segment


Providers

SOGS

PLPC

SPGF

XBAS

PXMF


Long Term Archive

CEC

IAT

MF

SPQC

Refer.Processor

Level 0

PDPC Core

Level 1

Level 2

Reprocessing Centre

Level 1

Level 2 MF

SPQC

Local Archive

Data Dissem.


Infrastructure

DPGS


USERS

ReportsData

Subscriptions

Data On-request

Datamat responsibility


SMOS Ground Segment


Providers

SOGS

PLPC

SPGF

XBAS

PXMF


Long Term Archive

CEC

IAT

MF

SPQC

Refer.Processor

Level 0

PDPC Core

Level 1

Level 2

Reprocessing Centre

Level 1

Level 2 MF

SPQC

Local Archive

Data Dissem.


Infrastructure

DPGS


USERS

ReportsData

Subscriptions

Data On-request

CNES responsibility


SMOS Ground Segment


Providers

SOGS

PLPC

SPGF

XBAS

PXMF


Long Term Archive

CEC

IAT

MF

SPQC

Refer.Processor

Level 0

PDPC Core

Level 1

Level 2

Reprocessing Centre

Level 1

Level 2 MF

SPQC

Local Archive

Data Dissem.


Infrastructure

DPGS


USERS

ReportsData

Subscriptions

Data On-request

ESA responsibility

Also:

Site

External network

GSOV

Operations


SMOS Data Flow

SPGF PLPC SOGS

XBASPDPC Core

L1OP

L2OP

SPQC

Instrument Planner CNES

Operations

Post-launch Support

Aux Data Handling

Aux Data Providers

Instrument plan

On-board SW updates

Telecommands

S Band uplink

X Band downlink

PXMF

Level 0 products

Auxiliary Data

SMOS Auxiliary

Data Files

Products and Aux Data

Quality Stamp

ESAC Operations

Expert Team

Reports


SMOS Data Flow

PDPC Core

L1OP

L2OP

SMOS Auxiliary

Data Files

ESAC Operations

Time Critical Users

Other Users

LTA

MF

IATs

Aux Data Generation

Expert Team

ESRIN Help Desk

Products, Reports and Aux Data via media

Products, Reports and Aux Data via

FTP

Reports

Products, Reports and

Aux Data

Distribution configuration


Reports

Maintenance Team

Problems identified

and corrections recommend

ed

Updated algorithms

and/or formats

Evolution Team

Improvements identified and

evolutions recommended


Reprocessing Campaign Data Flow

MMFI

SMOS Auxiliary

Data Files

ESRANGE Operations

Time Critical Users

Other Users

LTA

MFExpert Team


Reprocessed Products,

Reports and Aux Data via

FTP

Distribution configuration


Reports and Aux Data

Reports

Maintenance Team

Problems identified

and corrections recommend

ed

Updated algorithms

and/or formats

Evolution Team

Improvements identified and

evolutions recommended

SPQC

Quality Stamp

Level 1 Processing

Level 2 Processing

Reprocessing Centre

Reprocessed Products

Aux Data Handling

Aux Data Providers

Auxiliary Data


Reports and Aux Data via

media

User Services

Users

Product Orders

ESRIN Help Desk

Catalogue

Updates

Distribution Requests

Mission Manager

Reprocessing Campaign Authorisatio

n


DPGS

KO

LTA

CDR DPGS V1 DPGS V2

KO CDR OSAT

GSOV

Aug. 05 Oct. 06 June 07 Dec. 07

June 06 Feb. 07 Sept. 07

Dec. 06

Accepted SPGF and

PLPC

Accepted MF and SPQC

March 07 May 07

Accepted L1 (V1) and L2

(V1) Processors

Complete DPGS except: • L1OP = DPM of mid-2006• L2OP = prototype of late-2006• MF and SPQC = specific functionality of spring 2007

• L1OP = DPM of spring 2007• L2OP = prototype of mid-2007• MF and SPQC = specific functionality of late 2007

SMOS High Level Schedule


ESAC

Calibration Tower

You are here


ESAC


ESAC

North


ESAC

Calibration Tower

Future SMOS Antenna


SMOS Mask 2006-10-17

0.000

2.000

4.000

6.000

8.000

10.000

12.000

14.000

16.000

18.000

0.000 50.000 100.000 150.000 200.000 250.000 300.000 350.000

AZIMUTH (deg)

ELEV

ATIO

N (d

eg)

Assumes removal of the TS-1 antenna.

Castle

Trees

Calibration Tower (93.8 degrees)

Vil-1 antenn

a

Vil-2 antenn

a

Trees

Street light


ESAC Site

• OPS-1 ready; subsystem installation underway; furniture for early 2007

• Network installation underway; external access at end of 2006

PLPC-PXMF/SPGF

• Accepted at ESAC in October; ready for GSOV

XBAS

• Front end processing system from SMP very late; acceptance expected in November

• Antenna ready at Indra; Demeter tracking before year’s end; acceptance at ESAC expected in February

• FEP-AIT to be delivered to CASA in November for first SMOS space-to-ground compatibility testing

• Analysis shows that for full polarisation descending (evening) passes at ESAC are not sufficient for X-band download; some ascending (morning) passes will need to be used

Status


L1OP (Operational Processor)

CDR

L1PP (Prototype Processor)

V1 V2

Sept. 06 Jan. 07 Sept. 07

Oct. 06

DPM V1

V1 cross-verification

Level 1 Processor Status

V2 cross-verification

Dec. 06

TDS V1

TDD V1

DPM V2

TDS V2

TDD V2

April 07 Aug. 07June 07

Delta-CDR

April 07

• 8 dual processor dual core CPU machines required to meet 2.5 NRT requirement for dual polarisation• G/J matrix software from L1PP re-used• aux data handling in the CEC• consolidation of file formats

• regular updates available• on-going work on calibration issues, NIR processing,…


SM ESLs

KO DPGS V2 delivery

Dec. 06

Level 2 Processor Status

Algo validation: phase 1

• 8 dual processor dual core CPU machines required to meet 2.5 NRT requirement for dual polarisation• aux data handling in the CEC• consolidation of file formats versus the prototypes

• regular updates of prototype available• operational processor is largely re-use of the prototype• operational service from ECMWF for the end of 2006

OS ESLs

SM L2PP

OS L2PP

CDRAlgo validation: phase 2



July 06

LTA delivery

SM L2OP and open prototype

OS L2OP and open prototype

Apr. 07 Nov. 07


Description ESAC LTA Portable SMOS Product Quality Control (SPQC)

See slide X X

Monitoring Facility (MF) See slide X X

Interactive Analysis Tools (IAT) SMOS Data Viewer

(SDV) See slide X

Reference Processing Facility

Off-line processing environment for testing new processor releases before they go into operation

X

Level 1 Prototype Processor (L1PP)

Output of the development lead by Deimos Portugal

X

Soil Moisture Open Prototype (SM L2PP)

Output of the development lead by Array

X

Ocean Salinity Open Prototype (OS L2PP)

Output of the development lead by ACRI

X

SEPS SMOS end-to-end performance simulator

X

MIRASIM MIRAS instrument simulator for command and control testing

X

Land Cover Tool Overlays SMOS values onto Google Earth

X

Auxiliary Data Handling Generates aux data for the DPGS X IDL Tool-Box Standalone version of report generation

functions integrated into the MF X

Others TBD

CEC


SPQC

SW system that operationally controls the quality of all SMOS products and auxiliary data before they are released to their final users

KO CDR FAT V1 FAT V2

April 06 July 06 Mar. 07Nov. 06

ESA detailed specification of checks

Example checks resulting in no wide distribution to users:

• File internal consistency: size, structure, coherence

• Product confidence data generated by processor is bad

• Too many unrealistic values in product

• Product cannot be processed to next level

LTA and DPGS V2

DPGS V1


MF

SW system that 1) monitors performance of instrument, platform, and ground segment, and 2) performs automated data synthesis to help with calibration and product quality assessment

KO CDR FAT V0 FAT V3

April 06 July 06 Dec. 07Nov. 06

ESA detailed specifications

Example reports:• Daily monitoring of on-board temperatures• Daily monitoring of DPGS production versus planning• Monitor aux data usage• Monitor temporal and spatial distribution of anomalous values• Continuous collection of statistics per pixel• Monitoring of SMOS product values versus external sources• Monitor archiving and distribution completeness

DPGS V2 DPGS V1 and LTA

FAT V1

Mar. 07

FAT V2

July 07


SMOS Data Viewer

For all SMOS products and auxiliary data, the SDV 1) aids in the viewing of the contents, and 2) provides specific visualisation functions to help in the detailed analysis of the contents

KO CDR FAT V1

Mar. 06 June 06 Oct. 06

The SDV V1 version will be made available to all, along with a set of sample products, in early December.

FAT V2

Feb. 07

FAT V3

June 07

L1A, L1B only

Generic visualisation (scatter plot, temporal plot, spectrum plots, histograms)

Specific visualisation:• L1A: Correlator matrix (I, Q, amplitude and phase) for science and calibration products• L1B: Visibilities in the star domain• L1B: Fourier transformation of BT components in the star domain

L1C, L2 added

Specific visualisation:

• L1C: Projection on to a map (selection of incidence angle range)

• L2: TBD

Specific visualisation:

• TBD


SMOS Data Viewer


XML Read/Write API

XML R/W API

A first coherent set (XML R/W API, schemas, test data) will be available to all in December up to L1C. Level 2 should be available by March. All of these tools will evolve with the processor developments throughout 2007.

L0 Processor

L1 Processor

L2 Processor

MF

SPQC L2 Open Prototype

Aux. Data Handling

Test Data• subsystem/system acceptance

• algorithm verification/validation

File Formats• XML schema (BinX language for binary data sets)

• documents


Data Distribution

User Services

Data Dissemination

Eoli Interface (eoli.esa.int)• internet browser based ordering (L1B, L1C, L2 user, SMOS ECMWF only) of small volumes (<50 products)• intended for users new to SMOS

Help Desk• subscription configuration• on-call support during working hours

ESAC• all fast processing files• time critical users via network, everyone else via media• subscriptions: time range, geographic area, ascending/descending• likely >> 10 Mbps for SMOS

ESRANGE• all fast processing and reprocessing files• time critical users via network, everyone else via media• subscriptions: time range, geographic area, ascending/descending• likely >> 50 Mbps for SMOS

New UserTime Critical

UserNominal

User


Coming Attractions

ECMWF Data Assimilation Study• Prepare for the operational assimilation of SMOS data with a goal to reduce the start-up time

• Provide inputs to aid in SMOS validation activities

• KO by January with a 27 month duration

Surface TB Processor• On-going project with CLS will deliver an ATBD for spring 2007

• This project takes the ATBD and produces an operational processor that will go in the CEC at launch and in the operational fast processing chain by the end of commissioning

• Open ITT to go out in early 2007 with a KO by May


Coming Attractions

Reprocessing Study• Current processor implementation addresses only the fast processing scenario and not a reprocessing campaign: no allowance for improved aux. data or calibration processing

• Take current L1 and L2 DPMs and modify these to apply to the case of a reprocessing campaign

• Open ITT to go out in early 2007 with a KO by May and a completion at the end of 2007

• There will be a follow-on activity to implement this DPM in the Reprocessing Centre of the LTA for the first reprocessing campaign (expected to follow shortly after the end of the commissioning phase)


Coming Attractions

NRT Service• Decision will be made at the next PB-EO (November 23-24)

• Proposal is to acquire during each of the 10 non-ESAC orbits and deliver the data (one file per APID) to ESAC for processing using a separate NRT processing chain and then distribution to NRT users via the baseline ESAC data distribution infrastructure

• Procurement will be by two open ITTs: 1) acquisition of the 10 non-ESAC orbits and delivery to ESAC, and 2) implementation of a NRT processing chain largely independent of the existing DPGS infrastructure

• Not the optimal solution financially or technically but meets all the political constraints for acceptance


Summary

• It is subsystem acceptance and integration season and this will likely continue for the next four-six months

• The Fast Processing and LTA developments are both experiencing delays but this is no longer launch critical; a decision will have to be made in summer regarding which system is more likely to be ready and stable at launch

• Level 2 operational processor non-start is the biggest fire today

• CEC is starting to take shape; although granted it is yet to be visible outside ESA

• By next SAG there should be more to show you then these slides and some empty offices at ESAC

SMOS SAG, November 2-3, 2006 SMOS Data Processing Ground Segment (DPGS)

Documents

Transcript of SMOS SAG, November 2-3, 2006 SMOS Data Processing Ground Segment (DPGS)