Caio A. S. Coelho

Is a group of forecastsbetter than one?

Department of Meteorology, University of Reading

• Introduction• Combination in meteorology• Bayesian combination• Conclusion and future

directions

Plan of talk

Pierre-Simon Laplace(1749-1827)

C)yy(yy MSLSLS

The Pioneer of combination

xpya a

p

•

••

•

•

•

•

•

•

•

•• •

•• •

• xy

•

•

)1757,ichcovBos(SituationofMethod:y

squareLeast:y

MS

LS

))x(,x(CCwhere

Laplace (1818): Combination of two estimators

)xmin( 2

|)xmin(|

slope residual

+ numerical = methods

Have combined forecasts better skill thanindividual forecasts?

In modern times…

Forecasts from

•••

Forecast combination literature

• Trenkler and Gotu (2000): ~600 publications

(1970-2000)• Widely applied in Economics and

Meteorology

• Overlap of methods in these areas

• Combined forecasts are better than individual forecasts

Some issues

• What is the best method for combining?

• Is it worth combining unbiased forecasts with biased forecasts?

DEMETER coupled model forecasts

Nino-3.4 index (Y)

Period: 1987-99

9 members

Jul -> Dec

5 months lead

DEMETER web page: http://www.ecmwf.int/research/demeter

ECMWFMeteo-France (MF)Max Planck Institut (MPI)),(N~Y 2

tt

December Nino-3.4 raw forecasts

]ˆ96.1ˆ,ˆ96.1ˆ[:.I.P%95 tttt

Xt

tt

sˆ

Xˆ

),(N~Y 2tt

Forecast combination in meteorology

:wandw

:F

:M

:X

)t(Xww)t(F

io

M

1iiio

Linear combination of M ensemble mean forecasts X

constants

models

ensemble mean

combined forecast

• Bias-removed multimodel ensemble mean forecast (Uem)

• Regression-improved multimodel ensemble mean forecast(Rem)

• Regression-improved multimodel forecast (Rall)

Combination methods used in meteorology

Kharin and Zwiers (2002)

M

1iiio )t(Xww)t(F

How to estimate wo and wi ?

Kharin and Zwiers combined forecasts

2iii

n

1i

2i

2mRe

2mRe

)mReY(

n

1s

)s,m(ReN~Y

2iii

n

1i

2i

2Rall

2Rall

)RallY(

n

1s

)s,Rall(N~Y

]XXX[X

)s,Uem(N~Y

321

2X

Y: Observed December Nino-3.4 index

X: Ensemble mean forecast of Y for December

Thomas Bayes (1701-1761)

The process of belief revision on any event Y consists in updating the probability of Y

when new information X becomes available

The Bayesian approach

)xX(p

)Y(p)Y|xX(p)xX|Y(p

Example: Ensemble mean (X=x=27C)

Likelihood:p(X=x|Y)

Prior:p(Y)

Posterior:p(Y|X=x)

)C,Y(N~Y b

1TT

111T

obba

)SGCG(CGL

C)LGI()CGSG(D

)]YY(GX[LYY

)S],YY[G(N~Y|X o

Prior:

Likelihood:

Posterior:

1YYXYSSG

YGXGYo T

YYXX GGSSS

)D,Y(N~X|Y a

calibration

Bayesian multimodel forecast (B)

bias

July and December Reynolds OI V2 SST (1950-2001)

Nino-3.4 index observational data

Nino-3.4 indexmean values:Jul: 27.1CDec: 26.5Cr: 0.87

R2 =0.76

tyeartheforindex4.3NinoJuly

tyeartheforindex4.3NinoDecemberY

50.1C14.14),(N~|Y

t

t

1o

o2

t0t1ott

),(N~Y:iorPr 2otot t1oot

All combined forecasts

Uem

Rall

Rem

B

Why are Rall and B so similar?

Combined forecasts in Bayesian notationForecast

Likelihood Prior

Uem

Rem

Rall

B

)S,YY(N~Y|X o

)S],YY[G(N~Y|X o

)S],YY[G(N~Y|X o

)S],YY[G(N~Y|X o

)S,Y(N~Y YY

)S,Y(N~Y YY

),(N~Y 2otot

)C(Uniform

B priorRem and

Rall prior

)C,Y(N~Y b

)S],YY[G(N~Y|X oBayesian

combinationPrior:Likelihoo

d:

Skill and uncertainty measuresForecast MSE

(C)2

Skill Score(%)

Uncert.(C)2

Climatology 1.98 0 1.20

ECMWF (raw) 1.64 18 0.49

ECMWF (bias-removed) 0.18 91 0.49

ECMWF (regression-improved)

0.22 89 0.47

MF (raw) 0.27 86 0.39

MF (bias-removed) 0.22 89 0.39

MF (regression-improved) 0.20 90 0.46

MPI (raw) 15.24 -669 0.49

MPI (bias-removed) 1.52 23 0.49

MPI (regression-improved) 1.06 46 0.86

Uem 0.25 88 1.79

Rem 0.46 77 0.56

Rall 0.12 94 0.38

B 0.11 94 0.23Skill Score = [1- MSE/MSE(climatology)]*100%

Conclusions and future directions

• Forecast can be combined in several different ways

• Combined forecasts have better skill than individual forecasts

• Rall and B had best skill for Nino-3.4 example

• Inclusion of very biased forecast has not deteriorated combination

• Extend method for South America rainfall forecasts