Post on 15-Jan-2016
description
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Medium-range flood forecasting and warning
X General COSMO meetingCracow, 15-19 September 2008
Basin-oriented verification of
COSMO-LEPS system
Andrea Montani
ARPA-SIM Hydrometeorological service of Emilia-Romagna, Italy
Thanks to H. Asensio, R. Buizza, F. Pappenberger, B. Ritter, J.W.
Schipper
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Workpackage Medium-Range Plain Flood of PREVIEW project
Aims:1. Set-up and validation of the probabilistic medium-range flood
forecasting using the meteorological products for the Upper-Danube in the hydrological year 2002, which includes the large flood of August 2002; to achieve this,
• reruns of a number of state-of-the-art of atmospheric models were performed,
• “convenient” (basin-oriented) scoring techniques for probabilistic forecasts were developed.
2. In terms of operational applications, the goal is to demonstrate the usefulness of probabilistic medium-range flood forecasting as a sound basis for early warning and decision making.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
The region of interest
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Verification methodology
MAIN FEATURES:• Verification performed in terms of 24-hour
precipitation (from 6UTC to 6UTC);
• fcst ranges: 18-42h, 42-66h, 66-90h, 90-114h, …;
• observations: gridded observations (about 5 km of horizontal resolution) provided by JRC;
• verification domain: full upper-Danube river basin and 3 (out of 19) sub-basins;
• verification period: 20 July - 31 August 2002.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Sub-basin verificationID name of Basin area (km2) no. of gridded obs
1 Bratislava 131.978 5278
13 Wiblingen 2.247 80
16 Passau-Ingling 25.977 1045
20 Hofkirchen 47.534 1896
For each gridded observation, it was identified the sub-basin it belonged to.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Deterministic measures for verification
• Catchment-based measures• centre of gravity: it is a specific point at which the system's mass behaves as if it were concentrated. The centre of mass/gravity is a function only of the positions and masses of the particles of the system.
distmean
ccdistCoG obsf CoG: Centre of Gravity measure
cf: forecasted Centre of gravity dist: distance
cobs: observed Centre of Gravity distmean: mean distance of grid elements to catchment outlet
cf=
sNpo
ii
sNpo
iii
p
rp
int
1
int
1
ri are the geographical coordinates at grid point i
pi : forecast precipitation at grid point i
CoG (dimensionless) ranges from 0 to infinity … the lower the better (in practice the score is limited by the value 2, indicating a location error twice the mean catchment distance of the grid elements to the outlet).
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Ensemble systems in PREVIEW
– – – – VarEPS by ECMWF (global, ensemble)– COSMO-LEPS by ARPA-SIM (limited-area, ensemble)
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
CentreofGravity distance Aim: quantify the "location error" of the precipitation forecast (by the ensemble mean); the CoG is scaled with the "mean distance" in a catchment a value of cog=0.3 for the
catchment "Passau-Ingling" (mean distance =113 km) indicates a location error of the forecasted centre of gravity of precipitation in a catchment to the observerd centre of gravity of precipitation of about 30 km.
with the scaling, we get a dimensionless number (…the lower, the better…) to compare the absolute location error with respect to the catchment size.
CoG The absolute figures are very
low (distances of the order of a few tens of km).
Little dependence of the distance from the forecast range.
Slightly higher distances for the Passau-Ingling basin (possibly related to observation problems in the Alpine region).
Best results for the smallest basin.
CoG
dis
tan
ce
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Brier Score for VarEPS and COSMO-LEPS
BS
ECMWF : solid. COSMO-LEPS: dotted. Low values for both models:
GOOD! COSMO-LEPS performs better
over the smaller basins (not well resolved by VarEPS).
Similar performance of the two systems over the largest basin.
BS measures the mean squared difference between forecast and observation in probability space.
equivalent to MSE for deterministic forecast. BS between 0 and 1; the lower the better …. the largest (Bratislava) and the smallest (Wiblingen) basins are considered (tp > 80% of obs
distribution).
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
For the other results, come and see the poster!
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Thank you !
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
The 32-day unified VarEPS at ECMWF
Unification of the 15d VarEPS (50+1 members) and the 32d monthly forecast (MOFC) systems into the unified 32d VarEPS:
+768
15d and 32d VAREPS
T0 +240 +360 +768
NEW SYSTEM (since 11/3/2008)
Twice-a-day (at 00 and 12 UTC):
• d 0-10: TL399L62 uncoupled
• d 10-15: TL255L62 coupled at 00
Once a week:
• d 0-10: TL399L62 uncoupled
• d 10-32: TL255L62 coupled
old system
Twice-a-day (at 00 and 12 UTC):
• d 0-10: TL399L62 uncoupled
• d 10-15: TL255L62 uncoupled
Once a week:
– d 0-32: TL159L62 coupled
15d VAREPS
T0 +240 +360
T0
32d MOFC
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Skill of ECMWF predictions for hydrological modelling
Nash_Sutcliffe efficiency coefficient: normally used to assess the predictive power of hydrological models.
… the higher, the better (ranges from – to 1)
Timing of forecasts is good, although underestimation occurs.
The ensemble spread brackets observations only partially.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 200817th of July 2007
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
7 Panel version
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Atmospheric models in PREVIEW
– – – – – COSMO-LEPS by ARPA-SIM (limited-area, ensemble)
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
COSMO-LEPS (developed at ARPA-SIM)
MAIN FEATURES:• initial time: 12 UTC (i.e. once a day);• bc and ic: “selected” VarEPS members;• COSMO-LEPS configuration
– 10 members;– hor. res. = 10 km; 32 vertical levels;– forecast length: 132h;– archived variables: surf and plev up to +132h, every 3h;– output fields archived at ECMWF;
• rerun period: 20 July – 31 August 2002.
COSMO-LEPS: the Limited-area Ensemble Prediction System (LEPS), based on COSMO-model and implemented within COSMO (COnsortium for Small-scale MOdelling, including Germany, Greece, Italy, Poland, Romania, Switzerland).
integration domain
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Brier Skill Score vs forecast range BSS is written as 1-BS/BSref. Sample climate is the reference system. Useful forecast systems
if BSS > 0. BS measures the mean squared difference between forecast and observation in probability
space. Equivalent to MSE for deterministic forecast.BSS For low thresholds, better
performance over the smallest basin.
For higher thresholds, more difficult to assess a clear trend (possible sampling problems over small basins).
Worse than climatology only for d+5 range (10 and 20 mm threshold).
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Brier Skill Score vs increasing threshold BSS is plotted vs increasing threshold to assess the COSMO-LEPS skill as a function of rainfall
intensity.
BSS positive for all thresholds at both forecast ranges. At low thresholds, it is confirmed the higher skill of COSMO-LEPS over the smallest basin. Stable skill of the system for intermediate basins. For higher thresholds, more difficult to assess a clear trend (sampling problems over the small
basin).
BSS +90h
BSS +42h
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Ranked Probability Skill Score A sort of BSS, but “cumulated” over all thresholds. Useful forecast system, if RPSS > 0.
RPSS RPSS always positive.
Better (worse) performance of the system over the smallest (largest) basin.
Almost identical scores using either nearest grid-point (NGP) or bilinear interpolation (BILIN).
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
ROC area vs increasing threshold ROC area is plotted vs increasing threshold to assess the dependence of COSMO-LEPS HIT/FAR
diagram on rainfall intensity.
ROC area always above 0.7 for all thresholds at both forecast ranges. At low thresholds, it is confirmed the higher skill of COSMO-LEPS over the smallest basin (the
same as BSS). Scores increase with thresholds for intermediate and large basins. For higher thresholds, more difficult to assess a clear trend (sampling problems over the small
basin).
ROC +42h
ROC +90h
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
ROC area vs forecast range Area under the curve in the HIT rate vs FAR diagram. Valuable forecast systems have ROC area values > 0.6.
ROC ROC area always above 0.6; at low thresholds, better
COSMO-LEPS performance over the smallest basin;
possible sampling problems at 20 mm thresholds.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Outliers How many times the analysis is outside the forecast interval spanned by the ensemble
members. … the lower the better …
OUTL The absolute figures are quite
large (about 20% at d+5 range). Lower outliers for the smallest
basin. Similar percentages for all basins
at the longest forecast range.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Main results and future plans• Output files are of COSMO-LEPS, for the period 20/7/ to 31/8/2002, are
archived on ECFS (retrieval script disseminated) and ready to be used for hydrological purposes; for any kind of help with grib files, just ask.
• Verification against gridded observations (about 5300 in the full upper-Danube basin) indicates better performance of the system over the smallest basin, especially for low thresholds.
• OLD RESULT (Offenbach, 25-26-/9/2007): Verification against SYNOP stations (about 100 in the full upper-Danube basin) indicates slightly better performance of the system over larger basins, although results are not statistically robust.
• Difficulty to understand and use the CoG measure for a probabilistic system.
• Finish the work on the verification report.
• Provide COSMO-LEPS fields for 2-month real-time testing (no “bureaucratic” problems envisaged).
• Real-time testing of services will be done using the “improved” COSMO–LEPS (16 members; 40 vertical levels; physics perturbations).
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Main results
• During the hydrological year 2002, reruns of a number of NWP systems, both global-scale and limited-area, both deterministic and probabilistic.
• Development of new
– basin-oriented scores (e.g. centre of gravity);
– hydrological-oriented products (e.g. rivergrams);
• All systems seem to provide useful guidance for the possible occurrence of flood events also for forecast ranges up to 7 (5) days for global (limited-area) systems.
• Verification vs gridded observations (about 5300 in the full upper-Danube basin) indicates better performance of the high-res system over the smaller basin.
• The verification period is probably too short to draw general (and statistically solid) conclusions about the overall skill of the different forecast systems; the “Plain-flood campaign” clearly shows the potential of state-of-the-art NWP systems in the field of weather forecasting for river flooding.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Probabilistic measures for verification
N
kkk of
NBS
1
21
• Brier Score The BS is the mean-squared error of the probability forecasts:
where:
• the observation is either ok = 1 (the event occurs) or ok = 0 (the event does not occur);
• fk is the fraction of ensemble members which forecast a precipitation amount exceeding that threshold
• k denotes a numbering of the N forecast/event pairs.
BS ranges from 0 to 1, the perfect forecast having BS = 0.
BS is computed for a fixed precipitation threshold.
• ROC areaIt is the area under the Relative Operating Characteristics curve in the HIT rate vs FAR diagram
The integral under the curve is used to indicate the skill of the forecast.
ROC area ranges from 0 to 1 … the higher the better …
Useful forecast systems have ROC area values greater than 0.6
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Summary of measures
• Deterministic measures:• Root Mean Squared Error • Mean Error• Probability of detection• Probability of false detection• True Skill Statistics• Centre of gravity• Coverage
• Probabilistic measures:• Brier score• ROC area
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Performance measures
1) continuous measures (RMSE and MAE for the ensemble mean);
2) catchment-based measures (CentreofGravity for the ensemble
mean);
3) probabilistic measures (Brier Skill Score, ROC area, Percentage of
Outliers, …).Centre of Gravity: the centre of gravity of a system of raster cells is a specific point at
which, for many purposes, the system's mass behaves as if it were concentrated. The centre of mass is a function only of the positions and masses of the particles that comprise the system.
CoG: Centre of Gravity measurec: Centre of gravitydist: Distance f: forecastdistmean: mean distance of grid elements to catchment outletobs: observedp: precipitation
c= this score indicates a location error of the forecast, a perfect score having the
value of 0.
sNpo
ii
sNpo
iii
p
rp
int
1
int
1
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Catchment Information SystemRiver-grams
Sylvie Lamy-Thepaut, Enrico Fucile
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Main results
• During the hydrological year 2002 (1/10/2001 to 30/9/2002), reruns of a number of NWP systems, both global-scale and limited-area, both deterministic and probabilistic.
• Coordinated efforts to provide state-of-the-art weather forecasts over the Danube river basin.
• Development of basin-oriented scores
• Development of hydrological-oriented products (e.g. rivergrams)
• All systems seem to provide useful guidance for the occurrence of flood events.
increase in ROC area scores and reduction in outliers percentages; positive impact of increasing the population from 5 to 10 members (June
2004); some deficiency in the skill of the system can be identified after the system
upgrades occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels), BUT
scores are encouraging throughout DPHASE Operations Period.
Time-series verification scores indicate the following trends:
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
• “Close the gap” between hydrological and meteorological communities.
• Learn from MAP D-PHASE experience
• Exploit the wealth of information provided by probabilistic forecasts:
– assess performance over different domains (North and South of the Alps),
– study individual case studies,
– consider basin-by-basin performance.
• “Think about” increasing horizontal resolution to 7 km.
• Calibrate COSMO-LEPS fcsts using reforecasts (F. Fundel , Meteoswiss).
What is left?
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Sub-basin verificationID name of Basin area (km2) no. of “stations” approx no of grid points
1 Bratislava 131.978 5278 1320
13 Wiblingen 2.247 80 20
16 Passau-Ingling 25.977 1045 260
20 Hofkirchen 47.534 1896 470
For each gridded observation, it was
identified the sub-basin it belonged to.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Ranked Probability Skill Score A sort of BSS, but “cumulated” over all thresholds. Useful forecast system, if RPSS > 0.
RPSS RPSS always positive.
Better (worse) performance of the system over the smallest (largest) basin.
Almost identical scores using either nearest grid-point (NGP) or bilinear interpolation (BILIN).
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
CentreOfGravity distance Scaled (that is, dimensionless) distance between predicted (by COSMO-LEPS ensemble
mean) and observed centre of gravity for each catchment. the "cog" is an attempt to quantify the "location error" of the precipitation forecast; we have
scaled the "cog" with the "mean distance" in a catchment - so a value of cog=0.3 for the catchment "Passau-Ingling" with a mean distance distmean=113 km indicates a location error of the forecasted centre of gravity of precipitation in a catchment to the observerd centre of gravity of precipitation of about 30 km. With the scaling we get a (dimensonless) number to compare the absolute location error with respect to the catchment size.
… the lower the better …CoG
The absolute figures are very low (distances of the order of a few tens of km).
Little dependence of the distance from the forecast range.
Slightly higher distances for the Passau-Ingling basin.
Best results for the Wiblingen basin (the smallest one)
Similar distances for the other basins.
CoG
dis
tan
ce
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Catchment Information System River-grams
• Proto type, currently implemented operational.• Database of catchments will be extended to more
catchments in Europe and to include all major World catchments.
• Variables are currently static (always the same for all catchments), but will be ‘dynamic’ – reflecting the usage of catchments.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Outliers How many times the analysis is outside the forecast interval spanned by the ensemble
members. … the lower the better …
OUTL The absolute figures are quite
large (about 20% at d+5 range). Lower outliers for the smallest
basin. Similar percentages for all basins
at the longest forecast range.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Verification of GME forecasts against rain gauge data from high density observation network for the four sub-catchments of the Danube with different catchment sizes: • increasing centre of gravity score („location error“) with increasing forecast time;• a positive mean coverage error for this period.
Catchment-based verification of GME (2)
period 2002-01-01 – 2002-09-30
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
the verification period might be too short for scores to be statistically significant.
Verification of GME and COSMO-EU forecasts against rain gauge observations for the catchment Hofkirchen (upper Danube): • increasing centre of gravity score („location error“) with increasing forecast time • mostly a negative mean coverage error for this period
Catchment-based verification of DWD models (2)
period 2002-07-19 – 2002-08-20
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
COSMO-1km – cont. measures
A.Montani; Novelties in weather forecastingPREVIEW training workshop - Mosonmagyaróvár - 19-20 June 2008
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Dim 2
Initial conditions Dim 1 Dim 2
Possible evolution scenarios
Dim 1 Initial conditions
ensemble size reduction
Cluster members chosen as representative members (RMs)
LAM integrations driven byRMs
LAM scenario
LAM scenario
LAM scenario
COSMO-LEPS methodologyCOSMO-LEPS methodology
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
COSMO-LEPS (developed at ARPA-SIM)
• What is it?It is a Limited-area Ensemble Prediction System (LEPS),
based on COSMO-model and implemented within COSMO (COnsortium for Small-scale MOdelling, which includes Germany, Greece, Italy, Poland, Romania, Switzerland).
• Why?It was developed to combine the advantages of global-
model ensembles with the high-resolution details gained by the LAMs, so as to identify the possible occurrence of severe and localised weather events (heavy rainfall, strong winds, temperature anomalies, snowfall, …)
generation of COSMO-LEPS to improve the Late-Short (48hr) to Early-Medium (132hr) range forecast of
severe weather events.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
ROC area vs forecast range Area under the curve in the HIT rate vs FAR diagram. Valuable forecast systems have ROC area values > 0.6.
ROC ROC area always above 0.6. Similar results to those
obtained in terms of BSS: at low thresholds, better COSMO-LEPS performance over the smallest basin.
Possible sampling problems at 20 mm thresholds.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Probabilistic measures for verification (1)
The probabilistic measures we used, are the Brier Score and the ROC area. The thresholds for their computation are based on both absolute values (>1mm) and on the percentiles of the observed cumulative precipitation distribution (> 80% of observed).
N
kkk of
NBS
1
21
• Brier Score The BS is the mean-squared error of the probability forecasts; it averages the squared difference between pairs of forecast probabilities and the correspondent binary observations:
where:
•the observation is either ok = 1 (the event occurs) or ok = 0 (the event does not occur);
•k denotes a numbering of the N forecast/event pairs.
BS ranges from 0 to 1, the perfect forecast having BS = 0.
BS is computed for a fixed precipitation threshold and fk is the fraction of the ensemble members which
forecast a precipitation amount exceeding that threshold.
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Probabilistic measures for verification (2)
oN
oN
ca
aHR kk
kk
kk
)1(
)1(
oN
oN
cb
bFAR kk
kk
kk
• ROC areaIt is the area under the Relative Operating Characteristics curve in the HIT rate vs FAR diagram
Hit rate (HR) and false alarm rate (FAR) are computed for each probability class k:
where:
• the verification sample is subdivided into subsamples of size Nk, according to the probability with which the event is forecasted,
• ok is the frequency with which the event is observed, being forecasted with a given probability and is the sample climatology.
The cumulative HRk are plotted against the corresponding cumulative FARk, generating the ROC curve.
The integral under the curve is used to indicate the skill of the forecast.
ROC area ranges from 0 to 1, the higher the better.
Useful forecast systems have ROC area values greater than 0.6
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
COSMO-1km – Skill Scores (IMK)
A.Montani; Novelties in weather forecastingPREVIEW training workshop - Mosonmagyaróvár - 19-20 June 2008
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 200817th of July 2007
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Outline
• Introduction• Measure for model performance (precipitation
only)
• Atmospheric models re-run for PREVIEW:– GME by DWD (global, deterministic),– COSMO-EU by DWD (limited-area, deterministic),– VarEPS by ECMWF (global, ensemble),– COSMO-LEPS by ARPA-SIM (limited-area, ensemble),– COSMO-1km by IMK (limited-area, deterministic)
(tomorrow’s talk by J.W. Schipper)
• Application to the Danube sub-basins• Lesson learnt
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Deterministic measures for verification (1)
• Continuous measuresThe Mean Error (ME, bias) of the precipitation forecast against the measurement (F: Forecast; O:
Observations; N: sample size):
ME ranges from –infinity to infinity; the closer to zero, the better …
N
iii OF
NME
1
)(1
• Categorical measuresConsider thresholds (e.g. tp >1mm and tp > 80% of observed mean).
Scores are generated with the help of a contingency table:
o b s e r v e d
o b s e r v a t i o n > t h r e s h o l d
o b s e r v a t i o n t h r e s h o l d
f o r e c a s t > t h r e s h o l d h i t s f a l s e a l a r m s
F
orec
ast
f o r e c a s t t h r e s h o l d m i s s e s c o r r e c t n e g a t i v e s
P r o b a b i l i t y o f d e t e c t i o n misseshits
hitsPOD
; P r o b a b i l i t y o f f a l s e d e t e c t i o n
alarmsfalsenegativescorrect
alarmsfalsePOFD
T r u e S k i l l S t a t i s t i c POFDPODTSS ( u s e f u l f o r e c a s t f o r 0 < T S S < 1 … t h e h i g h e r t h e b e t t e r … )
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Atmospheric models in PREVIEW
– GME by DWD (global, deterministic)– COSMO-EU by DWD (limited-area, deterministic)– – –
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
GME: grid spacing: 40 km368642 * 40 grid elementstime step : 133 sec.forecasts up to 7 days
COSMO-EU: grid spacing: 7 km665*657 * 40 grid elementstime step : 40 sec.forecasts up to 78 hours
COSMO-DE: grid spacing: 2.8 km421*461 * 50 grid elementstime step: 25 secforecasts up to 21 hours
DWD operational NWP models
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
Example: DWD forecast of 24h accumulated precipitation with GME (global) and COSMO-EU (regional) for 2002-08-12 06 UTC to 2002-08-13 06 UTC, the forecast start time is 2002-08-11 12 UTC.
• forecasts with the global model GME for the
hydrological year 2002 (2001-10-01 2002-09-30).• forecasts with the high-resolution regional model
COSMO-EU for the PREVIEW special period
(2002-07-19 2002-08-20).
DWD meteorological forecasts (1)
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
24h catchment mean precipitation for the Hofkirchen catchment (upper Danube) forecasts with GME and COSMO-EU and corresponding adjusted radar data and high-density network rain gauge observations in the period 2002-08-01 to 2002-08-20.
There are differences between the two observing systems the overall evolution for the flooding event is well captured by both forecast models (COSMO-EU slightly better).
DWD meterorological forecasts (2)
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
GME forecasts vs rain-gauge data from high-density network for the four sub-catchments of the Danube with different catchment sizes: • positive bias for all four catchments in this period (the model “rains” too much);• positive TSS (true skill statistics), i.e. the forecasts are skilful.
Catchment-based verification of GMEperiod 2002-01-01 – 2002-09-30
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
the verification period might be too short for scores to be statistically significant.
GME and COSMO-EU forecasts vs rain-gauge observations for the catchment Hofkirchen:• negative bias for both models (the models “rain” too little); • GME slightly better for low thresholds• positive TSS (true skill statistics), i.e. the forecasts are skilful.
Catchment-based verification of DWD models
period 2002-07-19 – 2002-08-20
A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008
• “Close the gap” between hydrological and meteorological communities.
• Consider the outcome of MAP D-PHASE experimentation (model
performance may vary considerably from basin to basin).
• Develop and exploit “basin-oriented” calibration of NWP systems (being
implemented in these months for ECMWF varEPS and COSMO-LEPS).
• Exploit the wealth of information provided by probabilistic forecasts:
– get more and more acquainted with the concept of probability;
– consider the “gain” (€, $, Ft,…) presentation by D. Richardson.
What is left?