WWRP report to the CAS Management Group Gilbert Brunet WWRP/JSC Chair.
Forecast Verification Research Beth Ebert, Bureau of Meteorology Laurie Wilson, Meteorological...
-
Upload
brian-napier -
Category
Documents
-
view
213 -
download
1
Transcript of Forecast Verification Research Beth Ebert, Bureau of Meteorology Laurie Wilson, Meteorological...
Forecast Verification Research
Beth Ebert, Bureau of Meteorology
Laurie Wilson, Meteorological Service of Canada
WWRP-JSC, Geneva, 11-13 April 2012
2
Verification working group members
Beth Ebert (BOM, Australia)Laurie Wilson (CMC, Canada)• Barb Brown (NCAR, USA)• Barbara Casati (Ouranos, Canada)• Caio Coelho (CPTEC, Brazil)• Anna Ghelli (ECMWF, UK)• Martin Göber (DWD, Germany)• Simon Mason (IRI, USA)• Marion Mittermaier (Met Office, UK)• Pertti Nurmi (FMI, Finland)• Joel Stein (Météo-France)• Yuejian Zhu (NCEP, USA)
3
Aims
Verification component of WWRP, in collaboration with WGNE, WCRP, CBS
• Develop and promote new verification methods
• Training on verification methodologies
• Ensure forecast verification is relevant to users
• Encourage sharing of observational data
• Promote importance of verification as a vital part of experiments
• Promote collaboration among verification scientists, model developers and forecast providers
Relationships / collaboration
4
WGNE
SDS-WAS
YOTC
HyMeX
Subseasonal to Seasonal Prediction
Polar Prediction
WGCM
WGSIP
TIGGE
SWFDP
CG-FV
SRNWP COST-731
5
FDPs and RDPs
Sydney 2000 FDP
Beijing 2008 FDP/RDP
SNOW-V10 RDP
FROST-14 FDP/RDP
MAP D-PHASE
Severe Weather FDP
Typhoon Landfall FDP
6
SNOW-V10
• Nowcast and regional model verification at obs sites
• User-oriented verification
– Tuned to decision thresholds of VANOC, whole Olympic period
• Model-oriented verification
– Model forecasts verified in parallel, January to August 2010
• Status
– Significant effort to process and quality-control observations
– Multiple observations at some sites observation error
7
Forecast < 30 30 ≤ x < 50 50 ≤ x < 200 200 ≤ x < 300 300 ≤ x < 500 > 500 Total< 30 0 0 0 0 0 0 0
30 ≤ x < 50 0 0 0 0 0 0 050 ≤ x < 200 0 0 52 20 22 43 137
200 ≤ x < 300 0 0 76 18 19 103 216300 ≤ x < 500 0 1 26 15 12 60 114
> 500 0 9 831 246 170 3743 4999Total 0 10 985 299 223 3949 5466
lam1k Min. Visibility (m) at VOL HSS=0.095Observed
Wind speed verification(model-oriented)
Visibility verification(user-oriented)
8
FROST-14
User-focused verification
• Threshold-based as in SNOW-V10
• Timing of events – onset, duration, cessation
• Real-time verification
• Road weather forecasts?
Model-focused verification
• Neighborhood verification of high-resolution NWP
• Spatial verification of ensembles
Account for observation
uncertainty
9
Promotion of best practice
Recommended methods for evaluating cloud and related parameters
1.Introduction
2.Data sources
3.Designing a verification orevaluation study
4.Verification methods
5.Reporting guidelines
6.Summary of recommendations
10
Promotion of best practice
Verification of tropical cyclone forecasts
1. Introduction
2. Observations and analyses
3. Forecasts
4. Current practice in TC verification – deterministic forecasts
5. Current verification practice – Probabilistic forecasts and ensembles
6. Verification of monthly and seasonal tropical cyclone forecasts
7. Experimental verification methods
8. Comparing forecasts
9. Presentation of verification results
Verification of deterministic TC forecasts
11
Beyond track and intensity…
12
Model 2Model 1
Precipitation (MODE spatial method)
Track error distribution
TCgenesis
Wind speed
Verification of probabilistic TC forecasts
13
TIGGE ensembleintensity error before bias correction
After bias correction
Courtesy Yu Hui(STI)
Issues in TC verification
• Observations contain large uncertainties• Some additional important variables:
• Storm structure and size• Rapid intensification• Landfall time, position, and intensity• Precipitation• Storm surge• Consistency• Uncertainty • Info to help forecasters (e.g., steering flow)
• Tailoring verification to help forecasters with their high-pressure job and multiple sources of guidance information
• False alarms (incl. forecast storms outliving actual storm) and misses (unforecasted storms) currently ignored
• How best to evaluate ensemble TC predictions?
14
15
Promotion of best practice
Verification of forecasts from mesoscale models (early DRAFT)
1.Purposes of verification
2.Choices to be made
a. Surface and/or upper-air verification?
b. Point-wise and/or spatial verification?
3.Proposal for 2nd Spatial Verification Intercomparison Project in collaboration with Short-Range NWP (SRNWP)
16
Spatial Verification Method Intercomparison Project
• International comparison of many new spatial verification methods
• Phase 1 (precipitation) completed– Methods applied by researchers to same
datasets (precipitation; perturbed cases; idealized cases)
– Subjective forecast evaluations
– Weather and Forecasting special collection 2009-2010
• Phase 2 in planning stage
– Complex terrain
– MAP D-PHASE / COPS dataset
– Wind and precipitation, timing errors
17
Outreach and training
• Verification workshops
and tutorials– On-site, travelling
• EUMETCAL training
modules
• Verification web page
• Sharing of tools
http://www.cawcr.gov.au/projects/verification/
18
5th International Verification Methods Workshop Melbourne 2011
Tutorial
• 32 students from 23 countries
• Lectures and exercises (took tools home)
• Group projects - presented at workshop
Workshop
• ~120 participants
• Topics: – Ensembles and probabilistic forecasts
– Seasonal and climate
– Aviation verification
– User-oriented verification
– Diagnostic methods and tools
– Tropical cyclones and high impact weather
– Weather warning verification
– Uncertainty
• Special issue of Meteorol. Applications in early 2013
Seamless verification
19
Seamless forecasts - consistent across space/time scalessingle modelling system or blendedlikely to be probabilistic / ensemble
climatechange
local
point
regional
global
Sp
atia
l sca
le
forecast aggregation timeminutes hours days weeks months years decades
NWP
nowcasts
decadalprediction
seasonalprediction
sub-seasonalprediction
veryshortrange
"Seamless verification" – consistent across space/time scales
• Modelling perspective – is my model doing the right thing?
– Process approaches
• LES-style verification of NWP runs (first few hours)
• T-AMIP style verification of coupled / climate runs (first few days)
• Single column model
– Statistical approaches
• Spatial and temporal spectra
• Spread-skill
• Marginal distributions (histograms, etc.)
Perkins et al., J.Clim. 2007
"Seamless verification" – consistent across space/time scales
• User perspective – can I use this forecast to help me make a better decision?
– Neighborhood approaches - spatial and temporal scales with useful skill
– Generalized discrimination score (Mason & Weigel, MWR 2009)
– consistent treatment of binary, multi-category, continuous, probabilistic forecasts
– Calibration - accounting for space-time dependence of bias and accuracy?
– Conditional verification based on larger scale regime
– Extreme Forecast Index (EFI) approach for extremes
• JWGFVR activity
– Proposal for research in verifying forecasts in weather-climate interface
– Assessment component of UK INTEGRATE project
Final thoughts
• JWGFVR would like to strengthen its relationship with WWRP Tropical Meteorology WG– Typhoon Landfall FDP
– YOTC
– TIGGE
– Subseasonal to Seasonal Prediction
– CLIVAR
• “Good will” participation (beyond advice) in WWRP and THORPEX projects getting harder to provide– Videoconferencing
– Capacity building of “local” scientists
– Include verification component in funded projects
22
23
Thank you
Summary of recommendations for cloud verification• We recommend that the purpose of a verification study is considered carefully before commencing.• Depending on the purpose:
– For user-oriented verification we recommend that, at least the following cloud variables be verified: total cloud cover and cloud base height (CBH). If possible low, medium and high cloud should also be considered. An estimate of spatial bias is highly desirable, through the use of, e.g., satellite cloud masks;
– More generally, we recommend the use of remotely sensed data such as satellite imagery for cloud verification. Satellite analyses should not be used at short lead times, because of a lack of independence.
– For model-oriented verification there is a preference for a comparison of simulated and observed radiances, but ultimately what is used should depend on the pre-determined purpose. For model-oriented verification the range of parameters of interest is more diverse, and the purpose will dictate the parameter and choice of observations, but we strongly recommend that vertical profiles are considered in this context.
– We also recommend the use of post-processed cloud products created from satellite radiances for user- and model-oriented verification, but these should be avoided for model inter-comparisons if the derived satellite products require model input since the model that is used to derive the product could be favoured.
• We recommend that verification be done both against:– gridded observations and vertical profiles (model-oriented verification), with model inter-comparison done on a common latitude/longitude grid that
accommodates the coarsest resolution; – the use of cloud analyses should be avoided because of any model-specific "contamination" of observation data sets;– surface station observations (user-oriented verification).
• For synoptic surface observations we recommend that: – all observations should be used but if different observation types exist (e.g., automated and manual) they should not be mixed;– automated cloud base height observations be used for low thresholds (which are typically those of interest, e.g., for aviation).– We recognize that a combination of observations is required when assessing the impact of model physics changes. We recommend the use of
cloud radar and lidar data as available, but recognize that this may not be a routine activity.– We recommend that verification data and results be stratified by lead time, diurnal cycle, season, and geographical region.– The recommended set of metrics is listed in Section 4. Higher priority should be given to those labeled with three stars. The optional measures are
also desirable.– We recommend that the verification of climatology forecasts be reported along with the forecast verification. The verification of persistence
forecasts and use of model skill scores with respect to persistence, climatology, or random chance is highly desirable. – For model-oriented verification in particular, it is recommended that all aggregate verification scores be accompanied by 95% confidence intervals,
and reporting of the median and inter-quartile range for each score is highly desirable.
24