Verification of multi-model ensemble

forecasts using the TIGGE dataset

Laurence J. WilsonEnvironment Canada Anna Ghelli

ECMWF

www.ec.gc.caLawrence.wilson@ec.gc.ca

With thanks to Marcel Vallée

April 22, 2023

Outline• Introduction – TIGGE goals and

verification• Status of verification of TIGGE

ensembles– Standard methods– Spatial methods

• Precipitation verification project: plan and early results

• Summary• SEE the extended abstract also

April 22, 2023

Verification and the goals of TIGGE

• Goals:– Enhance collaborative research– Enable evolution towards GIFS– Develop ensemble combination methods; bias

removal• Essential question: If we are going to move

towards a GIFS, then we must demonstrate that the benefits of combined ensembles are worth the effort with respect to single-center ensembles. OR: Do we get a “better” pdf by merging ensembles?

• Verification – Relevant, user-oriented

April 22, 2023

Status of Verification of TIGGE ensembles

• Mostly model-oriented verification so far– Upper air data– Against analyses– Standard scoring and case studies

• Studies on the TIGGE website– Park et al, 2008

• First study involving several months of data• Found modest improvement with combined ensembles, greatest

benefits in tropics and lower atmosphere• “Advantage” of using one’s own analysis as truth

– Pappenberger et al.• Case study of flooding event in Romania• User-oriented, Q-Q plots, RPS and RMSE main scores used• Multimodel ensemble has best average properties, ECMWF next.

April 22, 2023

Studies using TIGGE data (cont’d)

• Johnson and Swinbank, 2008– Study of calibration/combination methods– Used only 3 ensembles– Mslp and 2m temperature, but from analyses– Multimodel ensemble improves on individual ensembles, but

not by much in general. More at 2m than 500mb• Matsueda 2008

– Comparison of 5 combined ensembles vs ECMWF alone– RMSE skill and RPSS with ECMWF as standard forecast– Multimodel eps outperforms ECMWF at medium and longer

ranges.

April 22, 2023

Status of TIGGE – related verification

• Current efforts – Verification of surface variables?– This conference ---?– Studies using spatial methods

• Ebert – application of CRA technique to ensemble forecasts. So far, only ECMWF.

• Application of Wilks minimum spanning tree or T. Gneiting’s multi-dimensional rank histogram for TC centers. (idea stage)

– Precipitation verification project:

April 22, 2023

Precipitation verification project

• Goal: to verify global 24h precipitation forecasts from all the ensembles in the TIGGE archive and combinations

• One region at a time, using highest density observations

• Canada and Europe so far• Methodology

– Cherubini et al upscaling, verify only where data available– Single station, nearest gridpoint where data is sparser– Kernel density fitting following Peel and Wilson to look at

extremes of distributions.

April 22, 2023

Precipitation verification project : methodology - Europe

• Upscaling:– 1x1 gridboxes, limit of model resolution– Average obs over grid boxes, at least 9 stns

per grid box (Europe data)– Verify only where enough data– Matches obs and model resolution locally– Answers questions about the quality of the

forecasts within the capabilities of the model– Most likely users are modelers.

April 22, 2023

European Verification

-Upscaled observations according to Cherubini et al (2002)

-OBS from gauges in Spain, Portugal, France, Italy, Switzerland, Netherlands, Romania, Czech Republic, Croatia, Austria, Denmark, UK, Ireland, Finland and Slovenia

-At least 9 stns needed per grid box to estimate average

-24h precip totals, thresholds 1,3,5,10,15,20,25,30 mm

-one year (oct 07 to oct 08

April 22, 2023

Reliability – Summer 08 – Europe – 42h

April 22, 2023

Reliability – Summer 08- Europe 114 h

April 22, 2023

Reliability – Winter 07-08 – Europe – 114h

April 22, 2023

ROC – Summer 08 – Europe – 42h

April 22, 2023

ROC – Summer 08 – Europe – 114 h

April 22, 2023

Precipitation verification project: methodology - Canada

• Single station verification– Canadian verification over

20 widely-spaced stations, only one station per gridbox; comparison of nearest gridpoint fcst to obs

– Pointwise verification, does not (we cannot) upscale properly because don’t have the data density necessary.

– Valid nevertheless as absolute verification of model predictions

April 22, 2023

Results – Canada – ROC curves – 24h

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

ROC Curve

False Alarm Rate

Hit

Rat

e

00.10.2

0.30.40.5

0.60.7

0.8

0.9

1

msc 0.825 ( 0.838 )

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

ROC Curve

False Alarm Rate

Hit

Rat

e

00.10.20.30.40.50.60.70.80.9

1

ecmwf 0.785 ( 0.842 )

April 22, 2023

Results – Canada – ROC Curves – 144h

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

ROC Curve

False Alarm Rate

Hit

Rat

e

00.10.20.3

0.40.5

0.6

0.7

0.8

0.9

1

msc 0.664 ( 0.66 )

0.0 0.2 0.4 0.6 0.8 1.00.

00.

20.

40.

60.

81.

0

ROC Curve

False Alarm Rate

Hit

Rat

e

00.10.20.3

0.40.5

0.6

0.7

0.8

0.9

1

ecmwf 0.703 ( 0.701 )

April 22, 2023

RMSE of pcpn probability – Canada – Oct 07 to Oct 08 – 20 stns

BOM in blue (darker blue); ECMWF in red; UKMET in greenCMC in gray; NCEP in cyan (lighter blue)

2.0 mm 10 mm

April 22, 2023

Verification of TIGGE forecasts with respect to surface observations – next steps

• Combined ensemble verification• Other regions – Southern Africa should be

next. – Non-GTS data is available• Evaluation of extreme events – kernel density

fitting to ensembles.• Other high-density observation datasets such

as SHEF in the US• Other variables: TC tracks and related surface

weather• Use of spatial verification methods• THEN maybe we will know the answer to the

TIGGE question.

April 22, 2023

www.ec.gc.ca

April 22, 2023

Issues for TIGGE verification• Use of analyses as truth – advantage of one’s own

model. Alternatives:– Each own analysis– Analyses as ensemble (weighted or not)– Random selection from all analyses – Use “best” analysis; eliminate the related model from

comparison– Average analysis (may have different statistical characteristics)– Model-independent analysis (restricted to data – rich areas, but

that is where verification might be most important for most users

• Problem goes away for verification against observations (as long as they are not qc’d with respect to any model)

April 22, 2023

Park et al study – impact of analysis used as truth in verification

April 22, 2023

Issues for TIGGE Verification (cont’d)

• Bias adjustment/calibration– Reason: to eliminate “artifical” spread in

combined ensemble arising from systematic differences in component models

– First (mean) and second (spread) moments– Several studies have/are being undertaken– Results on benefits not conclusive so far

• Due to too small sample for bias estimation?– Alternative: Rather than correcting bias,

eliminate inter-ensemble component of bias and spread variation.

April 22, 2023

ROC – Winter 07-08 – Europe – 42h

April 22, 2023

ROC – Winter 07-08 – Europe – 114h

April 22, 2023

Reliability – Winter 07-08 – Europe – 42h

April 22, 2023

Results – Canada – Brier Skill, Resolution and Reliability

Brier Skill and components - POP~6400 cases, 20 stns

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

1 2 3 4 5 6 7 8 9 10

Forecast day

BSS

, REL

and

RES

BSS - MSC

BSS - ECMWF

REL - MSC

REL - ECMWF

RES - MSC

RES - ECMWF