TIGGE Data Archive at NCAR

8th GIFS-TIGGE Working GroupWorld Meteorological Organization

Geneva22-24 February, 2010

Doug SchusterSteven Worley

Dave StepaniakOthers

Archive Summary

Online DataPeriod, most recent two weeks~ 5 TB , public products~ 1 TB, data preparation, subsetting~ 65 GB Tropical Cyclone Track Data (CXML)

Offline DataFull period of record~ 300 TB, NCAR MSS system

Provider Summary

Archive Status Service Improvements Special Projects Metrics Future Service Additions

Data Delivery (02/2010)

• Data exchanges using UNIDATA LDM, HTTP and FTP

• 440 GB/day (ECMWF = 332 GB/day) • 1.7+ million 2-D gridded fields/day

Archive Inhomogeneities

PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level

Variable LvL ECWF UKMO JMA NCEP CMA CMC BOM MetF KMA CPTC

Geopotential Z PL

Specific H PL

T PL

U-velocity PL

V-velocity PL

Potential Vor PT

Potential T PV

U-velocity PV

V-Velocity PV

U 10m SL

V 10m SL

CAPE SL

Conv. Inhib. SL

Land-sea SL

Mean SLP SL

Orog. SL

Skin T SL

Snow D. H20 SL

Snow F. H20 SL

Parameter vs Provider Matrix

Variable LvL ECWF UKMO JMA NCEP CMA CMC BOM MetF KMA CPTC

Soil Moist. SL

Soil T SL

Sunshine D. SL

Surf. DPT SL

Surf. ATmax SL

Surf. ATmin SL

Surf. AT SL

Surf. P SL

LW Rad. Out SL

LH flux SL

Net Rad SL

Net Therm. Rad SL

Sensible Rad. SL

Cloud Cov SL

Column Water SL

Precipitation SL

Wilt. Point SL

Field Cap. SL

PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level

Parameter vs Provider Matrix

2009 Service Improvements Improved Web Portal Performance

User data selection interface runs smoother.Automated capability to stage offline data for

download.

Added “Client for URLs” (cURL) based parameter subsetting capability.Directly extract selected 2D grids from files across

multiple data providers.Downloads managed from user’s workstation.Script templates are provided to drive “subscription

style” parameter download capability.

Strengthened data ingest processing.System monitoring (disk capacity, LDM status, etc)Failover mechanisms for LDM downtime.Data integrity checking.

2009/2010 Supported Special Projects

NCAR Research Applications Laboratory (RAL) Google African Meningitis Forecast InitiativeDr. Thomas Hopson Integrate weather and health data to manage African

meningitis outbreaks (in Ghana as starting point).Looking for 2-week lead time to position vaccines.

Processing ECMWF data from present backwards through the archive for model calibration.Total Precipitation, 2m Temperature, 2m Dew Point.Global Subarea 40 deg N to 40 Deg S.0.5 x 0.5 degree resolution.Assist RAL and Unidata in providing “Real-time” access

to TIGGE based Multi-Model Ensemble Forecast data.

http://www.ral.ucar.edu/projects/ghana/http://www.google.org/predict.html

NCAR TIGGE Full Model Data Archive Usage

YEAR GB AccessedGB

DownloadedFiles

DownloadedActive Users / Unique Users

2006 5 5 10 0 / 5

2007 900 900 32,000 10 / 50

2008 1,300 1,100 40,000 10 / 20

2009 13,700 1,900 68,000 10 / 30

2010 80 20 500 3 / 5

Total 15,985 3,925 140,510 33 / 80

cURL impact

NCAR TIGGE CXML Archive Usage

YEARGB

DownloadedFiles

Downloaded# Active Users /# Unique Users

2008 1 860 7 / 30

2009 26 12,300 17 / 35

2010 0.5 500 3 / 6

Total 27.5 13,660 27 / 81

Planned Service ImprovementsAdditional Online Storage

45 TB data online (3 months of most recent data).15 TB for data processing request output.Interface driven data processing and immediate

download capability extended from 2 weeks to 3 months.

Improved “Offline” Storage.Data being transitioned from current NCAR MSS

system to new HPSS system.Improved access speeds to offline data.Add data processing capability to offline data?

Expanded/Updated Computing.Improved LDM ingest reliability.Faster “turn-around” time on user sub-set requests.Potential for subscription services?

Potential Service Improvements Model Validation Data Portal (Pending

Funding)? Wide variety of observational and analyzed data

sources. In Situ Observation data (NCEP operationally collected data). Ocean based observations (ICOADS). Global tropical cyclone position and intensity data. Atmospheric Reanalysis (JRA, ERA, NNRR, CFSR). Global PCP datasets. Suggested TWG sets to support FDP’s. Will need discussions to set priorities for development.

Data sub-set and output format selection capability. Integrated With TIGGE model data portal

Data analysis tools and codes. Model Evaluation Tools offered by the Developmental Testbed

Center at NCAR

http://www.dtcenter.org/mrt/users

http://tigge.ucar.edu/