1

Malaquias Peña and Huug van den Dool

Consolidation of Multi Method Forecasts

Application to monthly predictions of Pacific SST

NCEP Climate Meeting, April 4, 2007

Acknowledgments: Suru Saha retrieved and organized the data, Dave Unger and Peitao Peng provided discussion to the subject

2

DATA• Forecasting tools: 8 CGCMs, 1 Statistical model

– NCEP CFS: 1981-2006, 15 membs, 9 leads– DEMETER : 1980-2001, 9 membs, 6 leads

• ECMWF• MPI• MF• UKMO• INGV• LODYC• CERFAX

– CPC’ Constructed Analog (CA) : 1956-2006, 12 membs,12 leads

This is what all have in common:• Monthly Forecasts, leads 0 to 5• Initial months: Feb, May, Aug, Nov• Length of retrospective forecasts: 21 years, 1981-2001

FOCUS: TROPICAL PACIFIC SST: 12.5 S TO 12.5 N

3

• Consolidation: Making the best single forecast out of a number of forecast inputs.

• Objective consolidation necessary as large supply of forecasts are available.

• If K is the number of participant forecast systems, ζ, predicting a particular target month with a given lead time, the consolidation is the following linear combination:

DEFINITIONS

K

iiiC

1

• For convenience, systematic errors and observed climatology are removed in ζ.

• The regression coefficients (weights), α, are based on past performance of the forecast system.

• o is the verifying field (e.g. observation; climatology removed). Suppose there are N cases of retrospective forecasts, then one can train a consolidation method by comparing:

NjCoK

i

ji

ji

jj ,..2,1,,1

4

OPTIMIZING WEIGHTS

• Find weights, αi ,for each forecasting tool, ζi, that

minimizes the (sum of square of) errors εj in

oZWhere Z is a matrix whose columns are the forecasting tools and rows are the data points in the training period, o is the column vector containing the verifying field, and ε is a vector of errors.

)()( ooSSE T ZZ

• Least square method (unconstrained regression):

oTTUR

ZZZ 1)(

5

eigenvalues Nino 3.4 PNA NAO

1 8.4584 5.9156 3.6889

2 0.1763 0.8394 1.402

3 0.1516 0.7808 1.1173

4 0.0707 0.42 0.8759

5 0.0536 0.3488 0.6316

6 0.0384 0.2874 0.5277

7 0.0297 0.1919 0.3978

8 0.0186 0.139 0.2462

9 0.0027 0.0772 0.1126

oZZZ TTUR

1 )(

1 2 3 4 5 6 7 8 9

0.482 0.2532 -0.5526 -0.5615 0.0189 0.0348 0.018 0.0381 0.0488

Corresponding weights for UR for lead 1, im 1

ILL-POSED MATRIX PROBLEM

1)( ZZ T

2i too large

6

RIDGE REGRESSION)()( oZoZSSE T

Constrained to: cT Minimize:

leads to

oZIZZ TTRID

1 )( Ridge Regression

11

K

oZIZZ TTRIM

)(

*1bIZZ TRIW

)(

(DelSole, 2007)

(ad hoc)

fob

iii 2

* 1

where and

K

i i

iiof

12

• Van den Dool estimates such that the weights are small and stable• Many more ways to find it• Depends on characteristics of covariance matrix ZTZ

7

RIDGE REGRESSION

• Model weights (αi, i=1..9) as a function of λ for three ridge consolidation methods.

• Figure illustrates asymptotic values. Our methods stop at λ=0.5.

• Unconstrained regression (λ=0) results in a wide range (including negative values) of weights.

RID RIM RIW

λλλ

8

CONSOLIDATION METHODS ASSESED

9

CROSS-VALIDATION

Anomaly Pattern correlation over the tropical Pacific. Average for all leads and initial months. Empty bar: Full (dependent), filled bar: 3-yr out cross-validated.

30

40

50

60

70

80

90

D1 D2 D3 D4 D5 D6 D7 CFS CA MM COR FRE RID RI2 RIM RIW UR

10

GRIDPOINT BY GRIDPOINT PERFORMANCE

11

EQUATORIAL PACIFIC

12

WESTERN TROPICAL PACIFIC

Trust in good models when performed well in a gridpoint. It goes to the

opposite direction of the bad models

13

WESTERN TROPICAL PACIFIC

MIXES CLOSEST NEIGHBORING GRIDPOINT

Trust in good models when performed well in a 3x3 box of gridpoints. It

goes to the opposite direction of the bad

models

14

WESTERN TROPICAL PACIFIC

Trust less good models, damps towards

climatology as negative weights are set to zero

DOUBLE PASS AND MIXES CLOSEST NEIGHBORING GRIDPOINT

15

INCREASING EFFECTIVE SAMPLE

1 GRIDPOINT BY GRIPOINT

2 3X3 BOXES3 5X5 BOXES 4 ALL GRIDPOINTS IN THE

DOMAIN5 GRIDPOINTS IN AND

OUT DOMAIN

Multi-methods average

AC

Skill of most consolidation

methods improve when effective sampling size

increases

Tropical Pacific SST. AC average for all leads and initial months

16

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

COR FRE RID RI2 RIM RIW

em 1X1

9m 3x3

9m all_gr

INCREASING EFFECTIVE SAMPLE Consistency: Percentage cases (leads and initial months) outperforming MM

17

RELATIVE OPERATIONING CURVES • Assess the ability to anticipate correctly the occurrence or non occurrence that SST anomalies will fall in the upper, middle and lower terciles.• Class limits defined by the observed SST during the training period • Probability information from the ensemble: counting the fraction of ensemble members that falls into the “above-normal”, “near-normal”, and “below-normal” categories, and interpreting this fraction as the probability that forecasts will fall in such categories. • Approach for the optimized weights: each ensemble member forecast is multiplied by normalized weights.

Lead 3

Upper tercile

Lower tercile

18

UPPER TERCILE

19

LOWER TERCILE

20

SUMMARYAll the points below are for the particular case of SST anomalies in the tropical Pacific. • Forecasts arising from a combination of multiple models of similar skill generally outperform those from individual models but not UR after CV-3.• Even the simple average of multi-methods shows consistent improvement over individual participant models.• Over all and after cross-validation, sophisticated consolidation methods marginally improve over the simple average.• Increasing the effective sampling size increases the skill and consistency of consolidation methods.• Consolidation methods improve significantly over the multi-methods average in the western Pacific.• Probabilistic assessment, as measured by ROC shows some improvement of consolidation methods over MM. Construction of the probability density function of the consolidation requires optimization.