Science Processing Status of the Earth Observing System Microwave Limb Sounder (EOS MLS) Elmain...

Science Processing Status of the

Earth Observing System Microwave Limb Sounder

(EOS MLS)

Elmain MartinezSeptember 27, 2010

Aura Science Meeting - Data System Working GroupBoulder, Colorado

Jet Propulsion LaboratoryCalifornia Institute of Technology

Copyright 2010 California Institute of Technology. Government sponsorship acknowledged.

Topics

27 September 2010 EOS MLS 2

• Recap MLS Interfaces & Dataflow• Data Processing Status• Software Status• Processing Environment Improvements• Summary

AuraAura

MLS

System Interfaces and Dataflow


US MLS Science Computing Facility (SCF)

Data Management Layer

UK MLS Science Computing Facility (SCF)

• Scientific Algorithm Development• Scientific Analysis• Science Processing SW Development• Quality Control & Assessment

All SIPS InputsAll Standard Products

DiagnosticsLogs

PGEsCalibration FilesProduction Policies

Science Investigator-led Processing System (SIPS)

Scheduling/Planning Layer

Data Management Layer

Fetch Input Data

Store Output Data

Product Generation Executables

(PGEs)

EOSDISCore

System (ECS)

Goddard Space Flight

Center (GSFC) Earth Science

(GES)

Data and Information

Services Center (DISC)

SDP Toolkit

Meteorological Data

Delivered Algorithm PackageData Quality Assessment

MLS L1B/2 PDRs

DN, PAN, PDRD

MLS L0 – PDSAura SC Engr.Aura SC Definitive OrbitAura SC AttitudeOperational Meteorological DataLeap Second & UTC Pole Data

DATA PROCESSING STATUS


MLS Near-Real-Time Status

• SIPS NRT configuration is continuing to deliver Ozone and Temperature L2 products using V2.23– Ingests pre-processed NRT MLS PDS data from the GES DISC

• 15 minute granules – L1 and L2 PGS software was modified in 2008 to improve product

delivery timeliness– L2 products are delivered to SCF and GES-DAAC; available ~3 hours

after observation• Customers include GMAO, NOAA/NCEP, NOAA/NESDIS/STAR,

Environment Canada, ECMWF, and Met Office

27 September 2010 5EOS MLS

Forward Processing Status

• SIPS is processing all incoming data with V2.23– L1/2 products delivered to SCF and GES-DAAC

• V2.23 in operation since March 2008– This is the 4th minor versions of V2

• v02.20 – initially released Nov 2006• v02.21 – de-convolve DACS radiances in Level 1• v02.22 – increased performance without change of quality • v02.23 – dealt with data variances produced by Aura anomaly

– (data since Dec 4, 2008)

• We will transition from V2.23 to V3.3 once the Data Quality Document is released


Reprocessing Status

• V3.10 reprocessing halted in Nov 2009 due to data quality concerns– Products delivered to GES DISC, but never publicly released; now removed

• SIPS is currently reprocessing all science data with V3.30– Reprocessing began in April 2010

• More than halfway through the 72 months (2273 days) to be reprocessed– V3.30 products being delivered to GES DISC; to be publicly released in Oct.


Day

s P

roce

ssed

SOFTWARE STATUS


Major Improvements in V3.3

• Software changes:– Processing with GEOS5 meteorology for dates after Sept 1, 2009

• MERRA data for dates prior

– Using HDF5 1.8.3 and SDP Toolkit 5.2.16– Products are augmented to be HDF5-NetCDF compliant– Software built with 64-bit compiler– Additional error handling added to scheduler– Products to be released to public once the Quality Document is finalized– See the poster “Version 3.30 Products from EOS MLS” for further details

• Science improvements– CH3CL (Methyl Chloride) is a new standard product

• See the Validation and Climatology poster on this species

– Revised spectroscopy to reflect best-fit to unknown continuum– Special handling of Core+R5 phase if THz is turned off


Version 4 Plans• Product Generation Software

– Target release date is Spring 2012– Improvements currently planned:

• Read various parameters using HDF5, e.g. temperature • Retrieve species in clouds• Flag radiances to not be used when clouds are detected• Generating a new product, CH3OH

– Intended to account for radiances seen in other species

– Integrate latest available version of HDF5 and SDP Toolkit

• NRT improvements:– XML-formatted metadata– Browse product for choosing granules to download– Capability for Data Casting (RSS)– Better forward model such as 2nd derivative


PROCESSING ENVIRONMENT IMPROVEMENTS


SIPS Processing Environment Improvements

• Replaced old IBM 32 bit production cluster with a Dell 64 bit cluster in June, 2010– Specs

• 48 nodes• 8 CPUs per node• 146 GB storage per node• 2.93 GHz• 24 GB memory (vs 16GB on older clusters)

• Benchmark tests for a single day completed 3 hours faster compared to an older cluster; ~27% improvement in speed– About 6.34 additional days per week with the 27% performance

improvement– With the additional cluster and its higher relative performance,

weekly output has increased by apx. 22 additional days per week


SIPS HW/Network Architecture


sips-switch (LAN)

Operations Dev/Test NRT

DB Server

sftp Server

SIPS Server

L1B (V2.23)

L2 (V2.23)

L2 (V3.30)

L2 (V3.30)

sftp Server

NRT Server (2)

JPL router

JPL net

L2 (V3.30)

Changes since 9-09:• Operations: - Replaced aging L2 Ops cluster - Replaced aging L1 server• Development - Replaced aging NFS/L1 server

L1B (V2.23)

DB Server

File Server/L1B (V3.30)

Dev/Test (2)

L2 (V3.30)

L1B (3, V30)

Replaced

SIPS Future Plans

• No upgrades planned for next FY• FY12 and beyond we will be looking at:

– Additional storage for V4 production– Replace aging servers– Replacing aging cluster


SCF Processing Environment Improvements

• Improvements in the last year:– Added 169 TB raw NetApp storage– Retired standalone NAS devices due to age.

• Moved data to the 60 TB NetApp

– Retired aging test cluster– MLS webpage upgraded

• Future plans:– No upgrades planned for next FY

• but there never seems to be enough storage…– FY12 and beyond we will be looking at:

• Purchase additional storage for V4 processing• Replace 32 bit cluster with 64 bit• Replace aging NetApp filer


SCF HW/Network Architecture


JPL Network

User Processing (4)

Web Server

NIS, Email (+backup)

DB Server (+ backup)

Collaboration Server

SCF Internal Network

Test Cluster

Dev Cluster

Network Storage224 TB

Touchscreen

Lab & Flight Ops (5)

Workstations (34)

Data Retrieval IF

Analysis Cluster

Expanded

Science Community Support

• Mobile Phone data tracking of MLS daily plots– Accessible from MLS webpage

• Hurricane portal (http://tropicalcyclone.jpl.nasa.gov)

– Tropical Cyclone Information System (TCIS) brings together satellite and in situ data sets from various sources (e.g. MLS, OMI, QuikSCAT, and others) so users can find information for a particular tropical cyclone.

• Site up since April 2010• 2005 data loaded, other years to follow• Site has an analysis tool that allows users to plot histograms, maps, and profiles for

many different data sets and products.• Uses MLS data products for temp, ozone, CO2, water vapor

• MLS webpage improvements, e.g. NRT Viewer– See the poster “The Microwave Limb Sounder Website: Improved Access to Data and

Information” for more information


SUMMARY


Recap of Future Plans• V4 is about 19 months away

– Algorithm cycle is usually 18 months or more, but it could be sooner if Level 2 algorithms can mature

• SIPS hardware is stable and should be for a couple years• SCF to make another leap to keep up…to purchase a 64 bit

development machine and additional storage in the future


Science Processing Status of the Earth Observing System Microwave Limb Sounder (EOS MLS) Elmain...

Documents

Transcript of Science Processing Status of the Earth Observing System Microwave Limb Sounder (EOS MLS) Elmain...