Huug van den Dool and Steve Lord

International Multi Model Ensemble

Two Consolidation Projects:

• Towards an International MME: CFS+EUROSIP(UKMO,ECMWF,METF)

11 slides

• Towards a National MME: CFS and GFDL, and NCAR/ CCM3.0/3.5 and NASA/GFSC

18 slides

Does the NCEP CFS add to the skill of

the European DEMETER-3 to produce a viable

International Multi Model Ensemble (IMME) ? Huug van den Dool

Climate Prediction Center, NCEP/NWS/NOAASuranjana Saha and Åke Johansson

Environmental Modeling Center, NCEP/NWS/NOAA

August 2007

DATA and DEFINITIONS USED• DEMETER-3 (DEM3) = ECMWF + METFR + UKMO

• CFS

• IMME = DEM3 + CFS

• 1981 – 2001

• 4 Initial condition months : Feb, May, Aug and Nov

• Leads 1-5

• Monthly means

DATA/Definitions USED (cont)

• Deterministic : Anomaly Correlation

• Probabilistic : Brier Score (BS) and Rank Probability Score (RPS)

• Ensemble Mean and PDF

• T2m and Prate

• Europe and United States

“ NO (fancy) consolidation, equal weights, NO Cross-validation”

Number of times IMME improves upon DEM-3 :

out of 20 cases (4 IC’s x 5 leads):

Region EUROPE EUROPE USA USA

Variable T2m Prate T2m Prate

AnomalyCorrelation

9 14 14 14

Brier Score

16 18.5 19 20

RPS 14 15 19.5 20

“The bottom line”

Frequency of being the best model in 20 casesin terms of

Anomaly Correlation of the Ensemble Mean

“Another bottom line”

CFS ECMWF METFR UKMO

T2m USA 4 5 5 6

T2m EUROPE 3 5 6 5

Prate USA 7 3 3 6

Prate EUROPE 11 0 0 5

Frequency of being the best model in 20 casesin terms of

Ranked Probability Score (RPS) of the PDF

“Another bottom line”

CFS ECMWF METFR UKMO

T2m USA 9 4 1 6

T2m EUROPE 9 3 4 3

Prate USA 19 0 0 1

Prate EUROPE 18 0 0 1

CONCLUSIONS

• Overall, NCEP CFS contributes to the skill of IMME (relative to DEM3) for equal weights.

• This is especially so in terms of the probabilistic Brier Scoreand for Precipitation

CONCLUSIONS (Cont)

In comparison to ECMWF, METFR and UKMO, the CFS as an individual model does:

• well in deterministic scoring (AC) for Prate and• very well in probability scoring (BS) for Prate

and T2m

over both USA and EUROPEAN domains

International Multi-ModelEnsemble (IMME) StatusS. Lord, S. Saha, H. Vandendool

Status• Goal: produce operational ensemble products from CFS

and EUROSIP seasonal climate products• EUROSIP

– ECMWF– Met Office– Meteo France

• Proposal will be submitted to EUROSIP Council– Covers

• Licensing and product distribution• Commercial interest and revenue sharing (none for US)

– Consistent with EUROSIP general provisions• Formal Memorandum of Understanding has been drafted

– Covers IMME products• Decision expected by end of calendar 2008

Status (2)• Some tenets of a potential agreement

– NCEP and E-partners will coordinate distribution of IMME products to their users on a regular monthly schedule

– Product delivery will not compromise any organization’s operational delivery schedules and commitments

– NCEP wishes to join the EUROSIP Steering Group as associate partner (non-voting member) and asks to participate in future meetings

– Associated research program possible for product improvement

M. Peña Mendez and H. van den Dool, 2008: Consolidation of Multi-Method Forecasts at CPC. Accepted JCLIM 2008

Unger, D., H. van den Dool, E. O’Lenic and D. Collins, 2008: Ensemble Regression. Accepted MWR

Wanqiu Wang: Pdf mapping methods

Apply to soil moisture analyses

We do work on methods!

Huug van den Dool, Yun Fan and Malaquias Pena

The Multi-Model Ensemble Approach for Soil Moisture Analyses in the Absence of Verification Data .

Suppose we want to do MME with EIGHT MODELS

• 1: R1• 2: R2• 3: NA(RR)• 4: ERA40• 5: LB Climate Divisions• 6: LB Global 0.5 degree • 7: Noah retroactive• 8: VIC retroactive

Common Period 1979-2001. Monthly mean total column soil moisture data on a 0.5 by 0.5 grid over the US.

We know how to take the mean, but how about a weighted mean??

Upfront we forgive models for:

• Error in the mean (most models much too dry in Illinois)

• Wildly different standard deviations

• CON is applied to standardized anomalies

K

CON = Σ α k w k (1)

k = 1

i.e. a weighted mean over K model estimates of standardized soil moisture anomalies.

One finds the K alphas, the weights, typically by minimizing the distance between CON and observed w for a number of cases.

What is a consolidation (CON)???

If we had observations for soil moisture we would first do a :

Classic or Unconstrained Regression (UR)

The general problem of consolidation consists of finding a vector of weights, α, that minimizes the Sum of Square Errors, SSE, given by the following expression:

SSE = (Wα - o)T(Wα - o) (2)

Then leads to WTWα = WTo

So the weights are formally given by

α = A-1 b (3)

where A = WTW is the covariance matrix, b=WTo and the superscript -1 denotes the inverse operation.

Equation (3) is the solution for the ordinary (Unconstrained) linear Regression (UR).

Essentially, ridging is a multiple linear regression with an additional penalty term to constrain the size of the squared weights in the minimization of SSE (2):

J = (Wα - o)T(Wα - o) + λ αTα (4)

Minimization of J leads to

α = ( A + λ I ) -1 b (5)

where I is the identity matrix, and , the regularization (or ridging) parameter, indicates the relative weight of the penalty term.

Similarities between the ridging and Bayesian approaches for determining the weights have been discussed by Hsiang (1976) and Delsole (2007). In the Bayesian view, (5) represents the posterior mean probability of α, based on a normal a priori parameter distribution with mean zero and variance matrix (σ2/λ)I, where σ2I is the matrix variance of the regression residual, assumed to be normal with a mean zero.

Dilemma

• Outside Illinois we don’t have (sufficient) soil moisture observations to consider CON methods. (Equal weight is always possible of course).

Line of Attack• In the absence of soil moisture data…we could use co-

located Temperature data (two months later) to do a CON (at least in ‘warm’ half of the year). This CON serves CPC’s application.

• In a sense we weigh models by their ability to predict co-located future temperature (April thru September only).

• As an aside: We know (and hope) that soil moisture also helps in non-co-located T&P, but we cannot easily work this into a weighting scheme. The local effect on T is undisputed (e.g. dry/wet soil leads to high/low temps – thus expect negative weights!)

• A hydrologist could do this against runoff obs, an agronomist against crop yields, disease (obs!) over matching years

LBcd LBgl R1 VIC ERA40 Noah R2 RR

Shown above is the vector bX100 in Eq.(3), which is also the correlation between each model’s soil moisture and the temperature two months later.

α = A-1 b (3)

Conclusions

1) All model’s w correlates negatively with future T. Good!

2) Some models (the 2 LBs, VIC, Noah ….) correlate a little better (with future T) than others (over 1979-2001)

3) A skill based weighting scheme without consideration of co-linearity would give the highest weights to these models (CPC ‘standard’)

4) Correlations (even -0.15) are modest, even if highly significant. Remember: This is an aggregate for all of the US and 6 warm months (April-Sept) combined.

-14.6 -14.9 -8.4 -13.5 -9.8 -13.3 -10.3 -10.0

LBcd LBgl R1 VIC ERA40 Noah R2 RR

-.035 -.082 .007 -.032 +.008 -.042 -.029 +.046

Conclusions

1) In the co-linear mix the Leaky Buckets carry most of the weight, followed by Noah and VIC etc. The remaining model speak for portions of the variance that, for the most part, are already accounted for by the leading models.

2) 75% ridging makes for a stable solution (all weights <=~0.)

Question:

1) How much better is the weighted average than an equal weight (-1/8th) mean?, and how much better than the best individual model???

Shown are the weights α calculated from Eq. (3), α = A-1 b with minimal ridging.

-.030 -.037 -.011 -.026 -.001 -.026 -.019 +.001 Ridge=0.75

Skill as measured by correlationX100.

CON 15.9

Equal Weight 14.8

Best single Model 14.9

Conclusions

1) Equal weight MME is NOT better than the best single model because it gives too much importance to poorly performing models.

2) Weighted MME is the best!, although the margin of gain may disappoint some of us.

UR MMA COR

RI RIM RIW

Climo

Classic

+Delsole equal weight limit

+CPC skill limit