WTF-CEOP Status and Plan “Data Mining” and “Data Integration” September 17, 2003

CEOS/WGISS, Chiang Mai – September 17, 2003 1

WTF-CEOP Status and Plan

“Data Mining” and “Data Integration”

September 17, 2003

Osamu Ochiai/NASDABen Burford/RESTEC

Toshio Koike/University of Tokyo


Contents

1. CEOP Data

2. WTF-CEOP (“Data Mining”)

3. The science

4. CEOP Data Integration

5. CEOP Interoperability

6. Way Forward


- CEOP -

Consolidated Enhanced Observation Period.• In-situ data (36 Reference Sites globally)• Satellite Data (200 Tbytes, subset and full scene)• Model Output Data (NWP from 10 Agencies)

EOP-1: 1 July 2001 thru 30 September 2001

EOP-2: 1 October 2001 thru 30 September 2002

EOP-3: 1 October 2002 thru 30 September 2003

EOP-4: 1 October 2003 thru 31 December 2004


CEOP In-situ Data


Satellite Data

• TRMM

• MODIS

• CERES

• AQUA• ADEOS-II• ALOS

• Subset scenes over Reference Sites• Full scenes over research areas• 200 TBytes

• ASTER

• MISR

• ENVISAT


Model Output Sources

Major National and Multi-National Centers BoM, CPTEC, ECMWF, ECPC, JMA, DAO, GL

DAS, NCEP, NCMRWF, UKMO

NWP – Numerical Weather Prediction data

Max Planck Institute for Meteorology (MPIM) at Hamburg, Germany


Resolution of the data (JMA)(example : Lindenburg)

288*145 (1.25 x 1.25 degree)3D - 23 pressure levels (fundamental variables)

640*320 (Native Gaussian, T213)2D (TOA, Surface processes )

320*160 (4 grids average )3D - 40 levels (heating rates)


Day N Day N+1 00 03 06 09 12 15 18 21 00 03 06 09 12 15 18 21 00

A A A A A A A A A

F F F F F F F F F F F F F F F F

NCEP Global Model Forecast (12-36 hour)

03 06 09 12 15 18 21 24 27 30 33 36

F F F F F F F F F

Repeat 12-36 hour forecast

NWP Assimilation and Forecast

Assimilation

Forecast

Forecast421.6 MBytes/day153.9 GBytes/year


Model Output Data (1/2)

shortwave downward fluxshortwave upward fluxlongwave upward fluxtemperaturepressurehumidityzonal windmeridional windpressure velocityhorizontal kinetic energy (KE)geopotential (gZ)cloud waterconvective latent heating ratestable latent heating rateconvective moistening ratestable moistening rateturbulent moistening rateturbulent heating rate

CEOP has requested 82 variables from the providers of Model data

short-wave heating ratelong-wave heating ratewater vapor zonal fluxwater vapor meridional fluxwater vapor vertical fluxwater vapor flux divergenceenergy (CpT+gZ+KE) zonal fluxenergy (CpT+gZ+KE) meridional fluxenergy (CpT+gZ+KE) vertical fluxenergy flux divergencelocal time tendency of temperaturelocal time tendency of KElocal time tendency of moistureLocal time tendency of cloudwatersurface pressureskin temperature2-meter temperature2-meter specific humidity

u-component at 10 mv_component at 10 mpotential temperature at 10 mspecific humidity at 10 msnow water equivalentSnow depthvegetation waterplanetary boundary layer heightshortwave downward fluxshortwave upward fluxlongwave downward fluxlongwave upward fluxsensible heating latent heatingSnow and frozen ground conversion to soil watermeridional wind stresszonal wind stress


10 m turbulent kinetic energy in a layerprecipitation (total)precipitation (snow)surface runoffbaseflowlocal surface pressure tendencylocal skin temperature tendencylocal snow water equivalent tendencysoil moisturesoil temperature

infiltration ratelocal soil moisture tendencylocal temperature tendencysubsurface temperatureground Heat Fluxbase flow runoffPrecipitation type 1rain or 2snowelevationSurface albedostation land/sea/ice mask 0(land)or1(sea)or(2)ice

Cloud coveraerosol concentrationsurface exchange coefficientroughness lengthVegetation coverWater table (wells)streamflowStream dischargeReservoir storage

CEOP is mapping these 82 variables to:•the standard GRIB list of variables•the variable names of model data•the variable names of in-situ data•the variable names of MOLTS•the variable names of satellite data

This mapping among variable names will be a key component of “Data Integration” for CEOP data!

Model Output Data (2/2)


Data Integration

“Data Integration”• Users can access all types of data in all locations • Models can be run on data from all areas.• Model output can be compared.• Common formats• Common data types• Consistent data variable names

What else can we do to expand “Data Integration”?


CEOP “Data Mining”(WTF-CEOP)

Validate Soil Moisture Algorithm• Input data - AMSR data (6.9 GHz, 18.7

GHz, 36.5 GHz)• Validation data - Soil Moisture in-situ data

First • Get the scenes of AMSR data over a

CEOP Reference Site (Mongolia)• Data for July, August and September, 2001


O UB

O M G

O C YM O N G O LIA SA

C H IN A

RUSSIA

0

1

2

3

4

5

6

7

8

A B C D E F G H I J

AWS - Automated Weather Station (6)

ASSH – Automatic Station for Soil Hydrology (12)

AMSR Data (6.9 GHz, 18.7 GHz, 36.5 GHz)

Soil Moisture In-situ Data

MongoliaReference Site


Two years of validation

Mongolia Reference Site/AMSR Data:

2 scenes/day x 365 = 730 scenes of AMSR data (times 3 frequencies = 2190 scenes)

2 scenes/day x 365 x 2 yr = 1,460 scenes of AMSR data (extract 17, 520 pixels from 114 GBytes of data)

36 Reference Sites: 52,560 scenes (4.1 TBytes of data)


WTF-CEOP User Interface

CEOP Data Integration Center“Data Analysis System”

User Interface

Select Location Select Time Range Reference Site From: 01 07 2001 From: 30 09 2001

Select Data In-situ Satellite MOLTS Model Output

Reference SiteTibetMongoliaHimalayasNSCSSJetc.

Satellite DataTRMM/PRTRMM/TMIADEOS II/AMSRADEOS-II/GLIetc.

In-situ DataPs (mb)Ts (degC)RH (%)Td (degC)Mv (%)Ws (m/s)etc.

AMSR Data 6.9 GHz 10.7 GHz 18.7 GHz 23.8 GHz 36.5 GHz 89.0 GHz


Data Mining Request

Prepare a “Distributed Data Mining Request”• Time Range – July 1 to Sep. 30, 2001.• AMSR data (6.9 GHz, 18.7 GHz, 36.5 GHz)• Soil Moisture in-situ data• Locations (12 sets of latitude/longitude values)

Format for Data Mining Request

(Use OGC Web Coverage Server (WCS) format?)


Data Mining Module3. Do data mining of Satellite Data.

User Interface Module1. User I/F (menus).2. Send Data Mining Request.

Final Module4. Do data mining of CEOP in-situ data.5. Send satellite and in-situ data results set to user.

NASDA

NASDA

NASDA

NASA ESA




NASDA, NASA, ESASatellite “Data Mining Module”

1. Catalog Search for all AMSR scenes over Mongolia (bounding box search) for July 1 to Sep. 30, 2001

• Use agency catalog search client (e.g. EDG)• Prepare a file with scene times and file names of AMSR data over

“Mongolia”

2. Processing - for each scene of AMSR data• Open file, geolocate pixels, extract 12 pixels over the Mongolia lat/

lon locations.

3. Send pixels, with the times of the satellite scenes, to the Final Module at NASDA.


Final Module

1. Extract in-situ data• Find time of satellite scene (e.g. July 1, 2001 at 01:05).• Find closest time of in-situ data (e.g. July 1, 2001 at 01:00).• Extract soil moisture values, at 12 ASSH locations, from

Mongolia in-situ data.

2. Put satellite scene time, satellite pixel values, in-situ data time and in-situ soil moisture values into Results File (in a convenient format).

3. Send Results to the user.


“Expand the Picture”

The Science Picture• Develop Algorithms

• Validate algorithms

• (Produce product – GRID?)

• “Virtuous Cycle” of algorithm development


Hydrology ResearchHydrology Research

Surface Emissivity & Temp.

Microwave Radiometer

Soil Moisture Snow

Precipitation

Land Surface Scheme

Snow Physics

Model

CloudPhysicsModel


Precipitation Algorithm


Water Cycle – Energy Budget

convLQconvHLEHRRe ssnt convergence/divergence

Air column

land surface

RnLEsHs

convH+convLQ

Rt

top

surface

vp

top

surface

vp dzLqTCdzLqTCe )()( 22221111 Dry Air-massEnergy

Water VaporEnergy

AMSRAMSRETMIMODIS(ASAR)(ETM)+4DDA

MODISGLICERESSRBISCCPERB

AIRSAMSUHSBSSM/T2HIRSGVAP

Top of atmosphere

RadioSondeProfilers

RadiationSensors

Eddy Corr.Bowen Ratio

Model Output


divLQPELtLW s )(/

divQ divQ

Land surface

Air column

EsP

Tropopause

convergence/divergence

Water Cycle – Water Budget

AIRSAMSUHSBSSM/T2HIRSGVAP

RadioSondeProfilers

AMSRAMSRETMISSM/IPRGPCPMODIS(ASAR)+4DDA

Rain GaugeRadar Eddy Corr.Bowen Ratio

Model Output

Pre-GPM


60

59

58

57

…………………………. …………………………. …………………………. …………………………. …………………………. …………………………. …………………………. …………………………. ………………………….

Geolocation and Temporal Integration

100 101 102 103


CEOP Data Integration

• Common formats• Consistent data types• Consistent variable names

With CEOP “Data Mining”• Single user interface (menus)

• Geolocated Integration (in-situ, satellite, model)

• Temporal Integration (in-situ, satellite, model)

• Delivery of exact pixels required (validation)


Add Interoperability

WGCV

• Land products for validation?

Core Site WTF

• Satellite or land products?

NOMADS, Earth System Grid, NERC Data Grid


Are we finished yet?

Problems? Yes!!Example – cloud cover.• Put cloud cover selection in menus (“No

scenes with cloud cover greater than xx%”)?• What cloud metadata do we have?• Cloud masks?• “Bad Data” values (e.g. 999.99 is bad data)• What is acceptable to scientists?• What can we do with the metadata?


Don’t Need Everything!

• Algorithm development

• Algorithm validation

• Climate Research model output validation (!!)

• “Data Integration”/”Data Mining”

+ GRID – Produce product, other?

+ WGCV – Data mining of land cover products?

+ Future missions - GPM

WTF-CEOP is enough! (For a great Plenary Demo)


Way Forward

1. Do we agree to support this process?2. Agencies determine resources.

– NASA, ESA, NASDA, Other – WTF-CEOP server!3. Develop a CEOP science scenario – CEOP scientists

+ CEOS team (choose sat. data to be supported)4. Science Scenario [balance] CEOS –

Types of data (MODIS is difficult, promote AIRS?)5. Schedule (CEOS Plenary in 2004?)6. Write a Project Plan7. Telecons/Detailed Schedule/Develop in stages?/ . . .

WTF-CEOP Status and Plan “Data Mining” and “Data Integration” September 17, 2003

Documents

Transcript of WTF-CEOP Status and Plan “Data Mining” and “Data Integration” September 17, 2003