The International TIGGE Archive Baudouin Raoult, Manuel Fuentes - ECMWF Steven Worley, Doug Schuster...
-
Upload
elizabeth-pearson -
Category
Documents
-
view
217 -
download
3
Transcript of The International TIGGE Archive Baudouin Raoult, Manuel Fuentes - ECMWF Steven Worley, Doug Schuster...
The International TIGGE Archive
Baudouin Raoult, Manuel Fuentes - ECMWFSteven Worley, Doug Schuster - NCAR
Bian Xiaofeng, Li Xiang - CMAMany Others!
3rd Thorpex Int. Science Symp., May 2009
Background: the TIGGE database
• Global ensemble forecasts to around 14 days generated routinely at different centres around the world
• Three archive centres: CMA, NCAR and ECMWF• Ten data providers:
– ECMWF, JMA (Japan), UK Met Office (UK), CMA (China), NCEP (USA), MSC (Canada), Météo-France (France), BOM (Australia), KMA (Korea), CPTEC (Brazil)
• Data exchanges using UNIDATA LDM, HTTP and FTP• 230 TB, 1.5 billion fields, growing by > 1 TB/week, 1.7 million fields/day
Background, Brief Time Line– March 2005, TIGGE Workshop at ECMWF
• Collect community input for TIGGE science and data requirements• Choose centralized archives for early start (ECMWF, NCAR, CMA)
– September 2005, meeting of technical experts at ECMWF• Representatives from ECMWF, NCAR, CMA, and NAEFS• Create technical implementation plan to address TIGGE needs
– November 2005, TIGGE Working Group at ECMWF• Present and get approval for the technical implementation plan
– October 2006, first EPS data is delivered– May 2006, User access to TIGGE archive starts– February 2008, all 10 international providers sending EPS
dataSignificant accomplishment, create 24x7 international
system in less than three years3rd Thorpex Int. Science Symp., May 2009
Quality assurance: homogeneity
• Homogeneity is paramount for TIGGE to succeed– The more consistent the archive the easier it will be to develop
applications
• There are three aspects to homogeneity:– Common terminology (parameter names, file names,…)– Common data format (format, units, …)– Definition of an agreed list of products (Parameters, Steps,
levels, …)
• What is not homogeneous:– Resolution– Base time (although most provider have a run at 12 UTC)– Forecast length– Number of ensemble
Strong governance
• Precise definition of:– Which products: list of parameters, levels, steps, units,…– Which format: GRIB2– Which transport protocol: UNIDATA’s LDM– Which naming convention: WMO file name convention
• Only exception: the grid and resolution– Choice of the data provider. Data provider to provide interpolation to
regular lat/lon– Best possible model output
• Many tools and examples:– Comprehensive documentation– Sample dataset available– Various GRIB2 tools, “tigge_check” validator, …– Scripts that implement exchange protocol
QA: Checking for homogeneity
• E.g. looking for wrong units?
Completeness
• The objective is to have 100% complete datasets at the Archive Centres
• Completeness may not be achieved for two reasons: – The transfer of the data to the Archive Centre fails– Operational activities at a data provider are interrupted
and back filling past runs is impractical
• Incomplete datasets are often very difficult to use• Most of the current tools used for ensemble
forecasts assume a fixed number of members from day to day
QA: Checking completeness (TIGGE history)
Archive Status and Monitoring, Variability between providers
Archive Status and Monitoring, Archive Completeness
PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level
Variable LvL ECWF UKMO JMA NCEP CMA CMC BOM MetF KMA CPTC
Geopotential Z PL
Specific H PL
T PL
U-velocity PL
V-velocity PL
Potential Vor PT
Potential T PV
U-velocity PV
V-Velocity PV
U 10m SL
V 10m SL
CAPE SL
Conv. Inhib. SL
Land-sea SL
Mean SLP SL
Orog. SL
Skin T SL
Snow D. H20 SL
Snow F. H20 SL
Archive Status and Monitoring, Archive Completeness
Variable LvL ECWF UKMO JMA NCEP CMA CMC BOM MetF KMA CPTC
Soil Moist. SL
Soil T SL
Sunshine D. SL
Surf. DPT SL
Surf. ATmax SL
Surf. ATmin SL
Surf. AT SL
Surf. P SL
LW Rad. Out SL
LH flux SL
Net Rad SL
Net Therm. Rad SL
Sensible Rad. SL
Cloud Cov SL
Column Water SL
Precipitation SL
Wilt. Point SL
Field Cap. SL
PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level
Access to TIGGE data
• Data is available to Research and Education after 48 hours– Self registration by agreeing to the terms and conditions
• Portals at CMA, NCAR and ECMWF
User Data Access and Support @ ECMWF
• TIGGE web site– Documentation– Model changes
• ECMWF portal offers:– Access to offline data– Aggregation along any axis (date, level, parameter, origin, ensemble,
…) – Provision of multi-model data on a single grid (regridding to any
lat/lon grid)– Sub-area selection– Reduces volumes to be downloaded by many order of magnitude
TIGGE @ ECMWF
TIGGE portal at ECMWF (using Ajax)
TIGGE Portal: grid selection
TIGGE Portal: area selection
TIGGE @ ECMWF
TIGGE @ ECMWF: Scheduling users’ requests
• Monitoring• Per user limits• Global limits
TIGGE @ ECMWF Batch access(recommended)
Perl or Python scripts
TIGGE@ECMWF: direct access to offline data (MARS)
User Data Access and Support, differences @ NCAR
• User data selection and subsetting equivalent to ECMWF
• 2.5 Weeks of Data Online• Archive organized in file groups by provider and
forecast initialization time– Pressure Level, Θ Level, PV Level, Single/Surface Level
• Multiple file download option – Direct through browser, using prepared ‘wget’ script
• Parameter download available through ‘curl’ scripts– Provides mechanism to build recurring requests
• Long-term file groups available by request
3rd Thorpex Int. Science Symp., May 2009
User Data Access and Support, differences @ NCAR
• Subset data format in GRIB2 or netCDF• Online queue to monitor subset data processing
progress• Subsetting completion email notification for users• TIGGE Research Support
– Library listing of software tools for data access and analysis– TIGGE Users Group web forum
• Community web space to share questions and findings• Provided by Unidata
• Archive of Tropical Cyclone (cxml) data for participating providers– Updated 6-hourly
3rd Thorpex Int. Science Symp., May 2009
User Data Access and Support, differences @ CMA
3rd Thorpex Int. Science Symp., May 2009
• Collaborated with ECMWF and CNIC
• Grid Middleware– Accessing the distributed
TIGGE data based on Grid middleware (GRIA, GOS) technologies
• Data retrieval and Derived Production Generation
• Request Monitoring and Query
• Access Control
User Data Access and Support, differences @ CMA
• Data Retrieval– Data Selection
• Selecting different data attributes, such as origin centre, date, base time, forecast time step, forecast type, level type, level and parameter.
• The different data options will be updated dynamically based on the user selection and on the constraints described in the data catalogue.
– Submit data request• If the number of requested fields is larger than the allowed limit, the
portal will suggest the users to split their data retrieval.
– Custom result products• Area selection• Grid conversion• Plotting
– Browse and download result products
User Data Access and Support, differences @ CMA
• Production Generation (e.g. ensemble mean)– Product selection
• Selecting the product and associated parameters based on the product catalogue• Submit product generation request
– In development, more products later
• Request Monitoring and Query– Check the real-time updates of the information about requests– Access the detailed monitoring information of requests for a period of time,
including request creation time, request finish time, process status, request condition, detailed processing information and result download.
– Query monitoring information according to the request date, request user id, request type and result status.
• Access Control– Several kinds of users (admin, data retrieval, product retrieval, …)
User Metrics at ECMWF
Active users
0
5
10
15
20
25
30
35
40
45
50
Dec-06
Jan-07
Feb-07
Mar-07
Apr-07
May-07
Jun-07
Jul-07
Aug-07
Sep-07
Oct-07
Nov-07
Dec-07
Jan-08
Feb-08
Mar-08
Apr-08
May-08
Jun-08
Jul-08
Aug-08
Sep-08
User Metrics at ECMWF : User Metrics at ECMWF
User Metrics at ECMWF : Data volumes
Data volumes (GBytes)
0
1000
2000
3000
4000
5000
6000
Dec-06
Jan-07
Feb-07
Mar-07
Apr-07
May-07
Jun-07
Jul-07
Aug-07
Sep-07
Oct-07
Nov-07
Dec-07
Jan-08
Feb-08
Mar-08
Apr-08
May-08
Jun-08
Jul-08
Aug-08
Sep-08
Retrieved
Delivered
Introduction to Poster Session demonstrationsInternational TIGGE Archive Centers
ECMWF http://tigge.ecmwf.int
NCAR http://tigge.ucar.edu
CMA http://wisportal.cma.gov.cn/tigge
Poster Sessions Demonstrations• When
• Tuesday, 15:00-16:30 (immediately following presentation)• Wednesday, 16:30-18:30• Any time, please ask Baudouin, Xiaofeng or Doug
• Purposes• Illustrate all features of the portals• Register users• Submit data requests• Demonstrate analysis tools
3rd Thorpex Int. Science Symp., May 2009
Credits
• Alex de Almeida Fernandes, Baudouin Raoult, Beth Ebert, Bian Xiaofeng, Dan Swank, Dave Stepaniak, David Ian Brown, Don Middleton, Dong-Joon Kim, Doug Schuster, Glenn Rutledge, Hannah Wilcox, Jean Nicolau, Jim Fraser, Joerg Urban, Julio Pablo Reyes Fernandez, Justin Cooke, Lang Hongliang, Lewis Poulin, Li Xiang, Ma Qiang, Manuel Fuentes, Michel Grenier, Okki Lee, Richard Dare, Richard Swinbank, Shintaro Yokoi, Simon Thompson, Steve Emmerson, Steve Worley, Steven Anthony, Tan Le, Tian Hao, Tom Yoksas, Waldenio Almeida, Yang Xin, Young-Youn Park, Yves Pelletier, …
• ... and many more