TIGGE Data Archive and Access TIGGE Data Archive and Access System at NCARSystem at NCAR
5th GIFS-TIGGE Working Group5th GIFS-TIGGE Working Group
South African Weather ServiceSouth African Weather Service
PretoriaPretoria
11-13 March 200811-13 March 2008
Steven WorleySteven Worley
Doug SchusterDoug Schuster
Dave StepaniakDave Stepaniak
-- Others ---- Others --
Topic OutlineTopic Outline
Archive SummaryArchive Summary Provider SummaryProvider Summary Improved Data Handling at NCARImproved Data Handling at NCAR NCAR’s portal and accessNCAR’s portal and access Next Steps - FutureNext Steps - Future
Archive SummaryArchive Summary
Online DataOnline Data– Period, most recent three weeksPeriod, most recent three weeks– ~ 3 TB , public products~ 3 TB , public products– ~ 6 TB, data preparation, subsetting~ 6 TB, data preparation, subsetting
Offline DataOffline Data– Full period of recordFull period of record– ~ 80 TB, NCAR Mass Storage System~ 80 TB, NCAR Mass Storage System
Provide access to bothProvide access to both
Provider SummaryProvider Summary
Geographic and methodsGeographic and methods MetricsMetrics
TIGGE Archive Centers and Data Providers
Archive Centre
Current Data Provider
NCAR NCEP
CMC
UKMO
ECMWFMeteoFrance
JMAKMA
CMA
BoMCPTEC
IDD/LDM
HTTP
FTP
IDD/LDM
Internet Data Distribution / Local Data Manager
Commodity internet application to send and receive data
Summary of Data Providers, March ‘08Summary of Data Providers, March ‘08
Center Conforming Parameters
Ens. Members
Model Res.
Fcst Length
Fcsts/ Day
GB/ Day
Fields/ Day
Files/ Day
ECMWF (ecmf)
70/73 51 N200 (Reduced Gaussian)
10 day 2 115 289,734 328
ECMWF (ecmf)
70/73 51 N128 (Reduced Gaussian)
10-15 day
2 24 138,978 160
UKMO (egrr)
70/73 24 1.25 x 0.83 Deg 15 day 2 21 175,680 488
JMA (rjtd)
61/73 51 1.25 x 1.25 Deg 9 day 1 7 113,192 74
NCEP (kwbc)
59/73 21 1.00 x 1.00 Deg 16 day 4 10 316,596 1040
CMA (babj)
60/73 15 0.56 x 0.56 Deg 10 day 2 28 72,510 82
CMC (cwao)
56/73 21 1.00 x 1.00 Deg 16 day 2 8 163,674 260
BOM (ammc)
55/73 33 1.50 x 1.50 Deg 10 day 2 8 147,972 164
MF (lfpw)
62/73 11 1.50 x 1.50 Deg
2.5 day
1 .15 7,558 33
KMA (rksl)
46/73 17 1.00 x 1.00 Deg 10 day 2 5 64,124 164
CPTEC (sbsj)
55/73 15 1.00 x 1.00 Deg 15 day 2 14 97,084 244
Total 22 240 1,587,102 3,037
Variable LvL ECWF UKMO JMA NCEP CMA CMC BOM MetF KMA CPTC
Geopotential Z PL
Specific H PL
T PL
U-velocity PL
V-velocity PL
Potential Vor PT
Potential T PV
U-velocity PV
V-Velocity PV
U 10m SL
V 10m SL
CAPE SL
Conv. Inhib. SL
Land-sea SL
Mean SLP SL
Orog. SL
Skin T SL
Snow D. H20 SL
Snow F. H20 SL
TIGGE Archive Completeness, 02/2008
PL = Pressure Level, PT = 320K Theta Level, SL = Single/Surface Level
Variable LvL ECWF UKMO JMA NCEP CMA CMC BOM MetF KMA CPTC
Soil Moist. SL
Soil T SL
Sunshine D. SL
Surf. DPT SL
Surf. ATmax SL
Surf. ATmin SL
Surf. AT SL
Surf. P SL
LW Rad. Out SL
LH flux SL
Net Rad SL
Net Therm. Rad SL
Sensible Rad. SL
Cloud Cov SL
Column Water SL
Precipitation SL
Wilt. Point SL
Field Cap. SL
TIGGE Archive Completeness, 02/2008
PL = Pressure Level, PT = 320K Theta Level, SL = Single/Surface Level
New Data Handling Hardware @ NCARNew Data Handling Hardware @ NCAR
Why ?Why ?– NCEP data processingNCEP data processing
Operational (non-TIGGE) grids via network (59 parameters)Operational (non-TIGGE) grids via network (59 parameters) Create TIGGE compliant grids, mainly metadata, some re-Create TIGGE compliant grids, mainly metadata, some re-
computation (e.g. precip.)computation (e.g. precip.) FTP upload 11 parameters from NCDC - FTP upload 11 parameters from NCDC - work in progresswork in progress Re-distribute to ECMWF and CMARe-distribute to ECMWF and CMA
– Data ingest from all providers, 240 GB/dayData ingest from all providers, 240 GB/day NCAR has maximum task, it produces nothing and receives NCAR has maximum task, it produces nothing and receives allall
– New security model at NCARNew security model at NCAR Forced by TeraGridForced by TeraGrid
– networked based sharing of data and computational cyclesnetworked based sharing of data and computational cycles CostCost
– $120K$120K
The Data Management ChallengesThe Data Management Challenges
Anomalies between centersAnomalies between centers Native horizontal resolutionNative horizontal resolution Number of ensemble membersNumber of ensemble members Number of forecast initialization times (1, 2, 4x daily)Number of forecast initialization times (1, 2, 4x daily) Forecast increment time periodForecast increment time period Forecast lengthForecast length Number of fields providedNumber of fields provided Internal file compression (e.g. jpg) was not specifiedInternal file compression (e.g. jpg) was not specified
Influence on ResearchInfluence on Research
User interface must inform users - commonalities constrain User interface must inform users - commonalities constrain researchresearch
NCAR TIGGE Portal NCAR TIGGE Portal
Address: Address: http:http://tigge//tigge..ucarucar..eduedu
Main FeaturesMain Features– Registration and LoginRegistration and Login– Get DataGet Data– User ToolsUser Tools– Documentation and LinkagesDocumentation and Linkages– Technical and Community Supported HelpTechnical and Community Supported Help
User Interface User Interface
Registration and LoginRegistration and Login– Required per international agreement Required per international agreement
Users electronically accept conditions for usageUsers electronically accept conditions for usage
– We capture metrics for We capture metrics for Name, email, organization name, organization type (univ., Name, email, organization name, organization type (univ.,
gov.,), and countrygov.,), and country Who , what, when files were downloadedWho , what, when files were downloaded
User Interface User Interface
Get Forecast DataGet Forecast Data Two Selection InterfacesTwo Selection Interfaces
– File GranularityFile Granularity Developed FirstDeveloped First
– Parameter GranularityParameter Granularity Added laterAdded later
Dates
Center
File Type
Forecast Time
Forecast Duration
Get Forecast Data
NCAR online file archive
• Selection options•Center(s)•Date•File type (sl, pl, etc)•Initialization time•Forecast length
Download Options• Point and click using browser, one file at a time• Script to run on local machine
•User and password encrypted ‘wget’ commands• background process to access all files
User customized files
• Selection options•Same as for files, plus•Parameter•Regridding•Spatial subsets•Formats, GRIB2, netCDF
Delayed ModeReal Time
Two User Interfaces
ToolsTools
Collection of tools with growing maturityCollection of tools with growing maturity– ContributorsContributors
NCAR NCAR ECMWFECMWF NOAANOAA UnidataUnidata
New API, GRIB2 to netCDFNew API, GRIB2 to netCDF– Developed as ECMWF/NCAR collaborationDeveloped as ECMWF/NCAR collaboration– Need to deploy in the NCAR systemNeed to deploy in the NCAR system
Tools, example; NCAR NCLTools, example; NCAR NCL
QuickTime™ and a decompressor
are needed to see this picture.
User HelpUser HelpTwo modesTwo modes Technical assistance directly from TIGGE staff at NCAR via Technical assistance directly from TIGGE staff at NCAR via
email email – Could originate from the portalCould originate from the portal
Open community website forum, including subscription emailOpen community website forum, including subscription email– Enrollees can post questions, give answers, and share ideas and Enrollees can post questions, give answers, and share ideas and
experiencesexperiences– Provided by UnidataProvided by Unidata
NCAR TIGGE archive usageNCAR TIGGE archive usage
1.1 TB, 95 K files1.1 TB, 95 K files 70 Unique data users70 Unique data users
User Profile
• A few users taking most volume
• 30 UU used parameter subsetting
• Most users take small amounts of data on single visit
Not sufficient for relevant research service
TIGGE data usage TIGGE data usage
Want public access to growWant public access to grow– Web advertisingWeb advertising– Helpful Idea - User Working Group meeting Helpful Idea - User Working Group meeting
in conjunction 3rd THORPEX Science in conjunction 3rd THORPEX Science SymposiumSymposium
THORPEX Exec. Com., 7th February Meeting.THORPEX Exec. Com., 7th February Meeting.
– Publication, possibly EOS/AGUPublication, possibly EOS/AGU– More presentationsMore presentations
Future @ NCAR, Phase I CompletionFuture @ NCAR, Phase I Completion
Finish new system installationFinish new system installation Verify and add NCEP parameters prepared at Verify and add NCEP parameters prepared at
NCDCNCDC Complete advanced subsetting featuresComplete advanced subsetting features
– Spatial, grid interpolation, and user selected output Spatial, grid interpolation, and user selected output format (GRIB2 and NetCDF)format (GRIB2 and NetCDF)
Continue data analysis tool developmentContinue data analysis tool development Quasi-automatic user access to long-term Quasi-automatic user access to long-term
TIGGE holdings from the NCAR MSSTIGGE holdings from the NCAR MSS
Future @ NCAR, add LAMSFuture @ NCAR, add LAMS
Focus on Americas and possibly one or two Focus on Americas and possibly one or two othersothers
Adapt access system for non-global domainsAdapt access system for non-global domains Much to be discussedMuch to be discussed
Future @ NCAR, Phase IIFuture @ NCAR, Phase II
Participation has limiting dependenciesParticipation has limiting dependencies– Finish Phase I Finish Phase I – Demonstrated need from research and education Demonstrated need from research and education
communitiescommunities– Requirements implied by Requirements implied by “data..accessible via
common web user interface”? Single user portal => uniform web service protocol Single user portal => uniform web service protocol
system-wide, interoperable across data, transport, system-wide, interoperable across data, transport, and searchand search
Track development, not contribute resources to R&DTrack development, not contribute resources to R&D
– Need funding path for all but very simple Need funding path for all but very simple adaptationsadaptations
WIS demonstrator project, WMO surplus funds? WIS demonstrator project, WMO surplus funds? THORPEX Exec. Com., 7th February Meeting.THORPEX Exec. Com., 7th February Meeting.
AcknowledgementsAcknowledgements
International data providers, excellent collaboratorsInternational data providers, excellent collaborators– TIGGE compliant data easily entered into the TIGGE TIGGE compliant data easily entered into the TIGGE
archive systemarchive system– Thanks for all the hard workThanks for all the hard work
International Archive Centers, excellent partnersInternational Archive Centers, excellent partners– Very prompt actionsVery prompt actions– Share much software/knowledge/experienceShare much software/knowledge/experience– Demonstrated a new level of International data provisionDemonstrated a new level of International data provision– NCAR has benefited NCAR has benefited
Users have been patient Users have been patient – This has been a Super-sized challengeThis has been a Super-sized challenge
NCAR staff are performing outstandinglyNCAR staff are performing outstandingly
Thanks to SAWS for hosting the 5th GIFS-TIGGE WGThanks to SAWS for hosting the 5th GIFS-TIGGE WG
Top Related