Evaluating density combinations -- Forecasting Norwegian ......Evaluating density combinations {...

Evaluating density combinations– Forecasting Norwegian GDP in real-time

Knut Are Aastveit Karsten R. Gerdrup Anne Sofie JoreChristie Smith Leif Anders Thorsrud

Presentation at the 6th Colloquium on modern tools forbusiness cycle analysis: “The lessons from global economiccrisis”, 26 - 29 September 2010

Context - Norges Bank’s system of averaging models (SAM)

1. Norges Bank instituted a project in 2006 to improve itsshort-term forecasts through model combination

2. Bjørnland et al. (2009) show that model combination improveforecasts from individual models, and that model combinationout-performs Norges Bank’s own short-term forecasts forinflation

3. SAM is used in the monetary policy process to provide theBank with model-based forecasts for GDP and inflation

4. SAM until September 2009:I Selection of the 8 best models for each horizon based on quasi

out-of-sample point forecast performance (RMSFE)I Forecasts are combined in a linear pool using univariate,

horizon-specific weights based on mean squared errors

5. SAM from September 2009:I Simplify the model space by grouping similar models together.I Two step combination procedure.

Economics Department/Nowcasting Team

Motivation

I Forecast combinations may produce better forecasts thanselecting, ex ante, a single model (portfolio diversification,unknown instabilities, and biases in individual models, seeTimmermann (2006))

I A number of studies have found that forecast combinationusing time-varying recursive weights, based on historicalforecast performance, is an ineffective strategy for improvingpoint forecast accuracy, see among other Stock and Watson(2004) and Clark and McCracken (2010)


Motivation

I Density forecasts provide policy-makers with a full impressionof forecast uncertainty

I Mitchell and Hall (2005) and Hall and Mitchell (2007) providea framework and justification for density combination

I Point forecasts are better seen as central points of ranges ofuncertainty

I Jore, Mitchell and Vahey (2010) examine density forecastsand conclude that adaptive weights improve simple weights

I Bache et al (2009) combines VAR and DSGE predictivedensities and Amisano and Geweke (2010) combines VAR,DSGE and Factor model predictive densities.


Contribution of this paper

What we do

I We produce combined density forecasts for NorwegianMainland-GDP from a system of different model classes.

I We compare our forecast combination approach with variouscombinations procedures.

I We use Norwegian real-time data.

ResultsI We show that both the log-score for the predictive densities

and the RMSE from our approach outperforms alternativestrategies:

I Selecting the best model ex-anteI Give all models equal weights


Outline

1. Modeling framework and data

2. Empirical exercise

3. Preliminary results

4. Concluding remarks


Forecasts

I We adopt an “expert” combination approach

I We define i = 1, . . . , N experts, where each expert producesone of the N density forecasts

I A decision maker combines the densities from two or moreexperts based on the fit of the density forecasts over theevaluation period


Forecast density combination

I The decision maker constructs the combined densities by alinear opinion pool method

p(Yτ,h) =N∑i=1

wi,τ,h g(Yτ,h | Ii,τ ), τ = τ , . . . , τ ,

where g(Yτ,h | Ii,τ ) are the h-step ahead forecast densitiesfrom individual model i, i = 1, . . . , N of a random variableY τ , (with realization yτ ), conditional on the information setIτ .

I wi,τ,h are a set of non-negative weights that sum to unity.


Choosing the weights

I Recursive logarithmic score weights (RLSW)

I We use the log score to measure the fit of the experts’densities through the evaluation period

wi,τ,h =exp

[∑τ−hτ ln g(yτ,h | Ii,τ )

]∑N

i=1 exp[∑τ−h

τ ln g(yτ,h | Ii,τ )] , τ = τ , . . . , τ

I Similarities with Bayesian approachI Approximate predictive likelihood approachI Bayesian Model Averaging


Two-stage modeling combination approach

Step 1

I Group all models into different model classes

I Models are combined within each model class using a linearopinion pool and recursive logarithmic score-based weights.

I Calculate the predictive densities for each model class

Step 2

I Combine the predictive densities from each model class intoone single combined density

I The different model classes are combined usingI Equal weightsI Recursive logarithmic score-based weights and linear opinion

poolI Optimal weights, see Hall and Mitchell (2007)


Model Classes

I VARs - Univariate and Vector AutoregressionsI Univariate, bi-variate and tri-variate autoregressionI 4 different lag lengths, 3 different transformations and 3

different estimation periodsI Number of models: 144

I SURV - Leading indicator and survey modelsI Bi-variate VARs with surveys and GDP growthI Business tendency surveys, Consumer confidence surveys,

Regional network surveysI Number of models: 46

I FM - Dynamic factor modelsI Dynamic factor models using monthly informationI 4 different factor combinations, 4 different lag lengthsI Number of models: 16


Empirical exercise

I Our recursive forecasting exercise is intended to mimic thebehavior of a policymaker forecasting in real-time.

I We use real-time vintage data for the Norwegian economy forall forecasts and realizations.

I Forecasts are performed at the last day of the quarter.

I Evaluation period is 2001q2 to 2010q1.

I We use both 5th release of GDP and the last available datavintage of GDP as “final” data for the evaluation.

I Compare the logarithmic score and RMSFE for 3 differentversions of the two-stage combination approach with

I Selection: Select the ex ante, best model measured bylogarithmic score.

I Equal weights: Pool all models together and give them equalweights.


Norwegian Mainland-GDP. Quarterly growth. Percent


Average recursive logarithmic scores - final vintage

(a) All Models (b) Model Classes


Average recursive logarithmic scores - 5th release vintage

(c) All Models (d) Model Classes


Recursive logarithmic scores ensembles - final vintage


Average recursive logarithmic scores - final vintage


Weights - Logarithmic Score - final vintage


Weights - Logarithmic Score - 5th release


PITS ensemble combination


Concluding remarks

What are the advantages of the two stage densitycombination approach?

I Performs well when forecasting GDP Mainland-Norway, bothin terms of point and density forecasts.

I The clustering facilitates story-telling – linking forecasts toparticular data

I Make it computationally possible to calculate optimal weightsI Ongoing extensions:

I Explore the importance of new data releases in real-time forNorwegian and U.S. data

I Investigate the importance of data revisions


Fanchart published after interest rate meeting


Density forecasts and outcomes


Evaluating density combinations -- Forecasting Norwegian ......Evaluating density combinations {...

Documents

Transcript of Evaluating density combinations -- Forecasting Norwegian ......Evaluating density combinations {...