TIGGE Archive Access at NCAR

Steven WorleyDoug SchusterDave StepaniakHannah Wilcox

Background on TIGGE

WMO World Weather Research Programme THORPEX

THe Observing system Research and Predictability EXperiment

THORPEX Interactive Global Grand Ensemble (TIGGE) Archive supports researchGrand Ensemble = multiple NWP centers ensembles are

combined (an ensemble of ensembles) 10 international NWP Centers contributing to TIGGE

Background on TIGGE

Three mirrored archive centers NCAR ECMWF CMA{Shared System Development!}

Daily Data Flow Metrics245 GB1.6 Million gridded fields as separate data packets

Data Receipt

Archive Centre

Current Data Provider

NCAR NCEP

CMC

UKMO

ECMWFMeteoFrance

JMAKMA

CMA

BoMCPTEC

IDD/LDM

HTTP

FTP

Unidata IDD/LDMInternet Data Distribution / Local Data ManagerCommodity internet application to send and receive data

NCDC

Major ChallengesInsure data receipt, build complete archive

Collate data fields into different files typesExchange manifest files as part of IDD/LDM data

transmission between Archive centersVerify send, receiveAutomated resend requests for missing fields

Harvest and hold metadata in MySQL DB’s Identify location of every field in file setUpdated often Critical for users interface and background data

processing

Major Challenges Limited online storage – 4 TB, ≅ 2.0 weeks

temporal coverageFull archive on NCAR Mass Storage System

User registration and metrics requiredAccept data policy; for research and education only48 hour delay from forecast initialization time

Major Challenges Access system must accurately display what

data are available as users make selectionsDriven by multi-center research (Grand Ensemble)E.g., want to experiment with NCEP and JMA

forecasts for 500 hPa temperatures Maximum common data is determine by JMA

12Z forecast initialization and 9-day forecast

Differences between centers

User access demonstration

Animation, what you will seeMultiple centers

(ECMWF, UKMO, NCEP, CMA, CMC, KMA)Fields/Parameters

(Geopotential Height, 2m Temperature)Levels

(500 hPa, Single Level)Spatial and temporal ranges

(Global, 3-days, 12Z initialization, 48 hour forecasts)Regridding to common spatial resolution

(1.5°)Output format

(netCDF)

Sample Data Request for an Event

Retrieve Completed Subset

Subset Request Animation

Gustav/Hannah Animation

Lessons Learned

Manifest files and automated resend are critical for a complete archive

The impact of different contributions from the NWP centers across archive cannot be under estimated

There are important design considerations to insure prompt browser interactions Caching data from the DB

Lessons Learned

Computational resource requirements ramp up quickly with multi-dimensional problemsD’s, center, ensemble member, parameter,

forecast length, etc. Archive file structure choices greatly impact

subsetting abilityTIGGE currently based on synoptic orderTime-series by parameter could be better?

End

http://tigge.ucar.edu/

Future

Consider adding storage resources to keep a longer period of data online

Integrate more computation resources to reduce time to fill requests

Create a way to re-stage MSS data to online for internet users

Outline

Brief Background on TIGGE Archive Data Receipt Challenges Access Lessons Learn

User registration and metrics required

Conditions for useResearch and education only48-hour delay on availability

Limited online storage – 5 TB, ≅ 2.5 weeks

1-2 TB additional storage for user request preparation

Supplement AccessDirect Archive File download, though interface or

Research Data Archive (RDA)All data (200 TB) available through RDA from

NCAR Mass Storage System TC data?