Why Forecasts Differ and Why are they so Bad? Roy Batchelor Professor of Banking and Finance, Cass...

Why Forecasts Differand Why are they so Bad?

Roy BatchelorProfessor of Banking and Finance, Cass Business School

But back in the real world…

• Forecasters often look (and feel) stupid

• This is becauseSometimes they can’t help it. Sometimes they deliberately make biased forecasts.

• Aim is to separate these causes of error, using evidence from panels of forecastersConsensus EconomicsBlue Chip Financial Forecasts

• Payoff – which forecasters are worth listening to, and when?

Plan of lecture

• Insights from recession forecasts

• Reasons for biased forecasts

• Who is most biased?

Forecasting the 1991 recession

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1991 1991 1991 1991 1991 1991 1991 1991 1991 1991 1991 1991

Consensus

Mr. Brightside 1

Dismal Scientist 1

HMT

-5

-4

-3

-2

-1

0

1

2

3

4


2008 2008 2008 2008 2008 2008 2008 2008 2008 2008 2008 2008 2009 2009 2009 2009 2009 2009 2009 2009 2009 2009 2009 2009

Consensus

Mr. Brightside 2

Dismal Scientist 2

HMT

Forecasting the 2009 recession

Should we privatise GDP forecasts?

• “Ageing model leads Treasury astray” (Sunday Times 9/2/92)

• “Time to take forecasting away from the Treasury” (Sunday Times, 12/4/92).

• What do you think?

Forecasts for 1990

-1

-0.5

0

0.5

1

1.5

2

2.5

3


1989 1989 1989 1989 1989 1989 1989 1989 1989 1989 1989 1989 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990

Consensus

Mr. Brightside 1

Dismal Scientist 1

HMT

Forecasts for 2008

-0.5

0

0.5

1

1.5

2

2.5

3


2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2008 2008 2008 2008 2008 2008 2008 2008 2008 2008 2008 2008

Consensus

Mr. Brightside 2

Dismal Scientist 2

HMT

Forecasts for 2010: … it’s too soon to know

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2


2009 2009 2009 2009 2009 2009 2009 2009 2009 2009 2009 2009 2010 2010 2010 2010 2010 2010 2010 2010 2010 2010 2010 2010

Consensus

Mr. Brightside 2

Dismal Scientist 2

HMT

Recession forecasts in general

• Loungani, P, 2001, How accurate are private sector forecasts? Cross-country evidence from consensus forecasts of output growth, International Journal of Forecasting, 17, 419-432

Stylised facts about Forecasts

• It’s hard to forecast recession. These are rare events, always with different causes.

• Consensus forecast usually starts close to the average growth rate, and then adjusts

• Accuracy-improving information arrives only about 12-15 months in advance of the end of the target year.

• Individual forecasts are distributed above and below the consensus in a consistent way. There are persistent optimists and pessimists.

Plan of lecture




Reasons For Bias

• Incompetence (unlikely, really…)

• Bias in the Consensus:

Learning about Structural Breaks

Market Incentives for Bias

• Bias in Individual Forecasts

Market Incentives for Product Differentiation

Learning about Structural Change

• Important

• Many countries (Japan, Germany, Italy, France) experienced slowdown in trend growth

• Optimal forecast will be biased to optimism as forecasters learn about the new trend

• Evidence supports this - forecasts in US, UK less biased

US – Log(GDP) and Forecast (Non-) Bias

Italy – Log(GDP) and Forecast Bias

Market incentives for bias

• Important for Investment Analysts

• Bias to optimism in earnings forecasts can arise from

Selection of firms/ sectors you believe inRelationship buildingTrade generation

• However, regulatory changes (Sarbanes Oxley) can change incentives…

• May also apply to some government forecasts

Financial analysts: Bias to optimism 1990-2003

Financial analysts: Bias to pessimism 2003-

Dispersion v Herding in Individual Forecasts

• There is evidence of herding – excessive convergence on the consensus – for financial analysts

• However, there is evidence of the excessive dispersion of forecasts by economic forecasters, who

underweight information in the consensus forecast (Batchelor and Dua, 1992, J Forecasting )

Maintain consistently optimistic or pessimistic priors from year to year (Batchelor and Dua, 1990, Int J Forecasting; Batchelor, 2007, Int J Forecasting)

Herding

• Financial forecasters have incentives to overweight consensus forecasts:

“Information cascade”: forecasts are made sequentially, so each forecast becomes part of the next forecasters prediction set. In aggregate published forecasts are biased towards the early forecasts.

“Incentive concavity”: rewards for an accurate “bold” forecast are smaller than penalties for an inaccurate bold forecast. Less experienced forecasters herd more since career prospects are at stake.

Dispersion and Product Differentiation

• Hotelling “location model” – don’t set up your store right next to everyone else, but don’t be too far out of town. So profit maximising strategy is to shade forecasts consistently away from the Consensus (even if you believe it is the best forecast)

• Batchelor and Dua (1990), Batchelor (2007) find individual forecasters persistently make optimistic or pessimistic forecasts relative to the consensus. Interpreted as an attempt to differentiate their product, increase press coverage, book sales, speaking fees etc.

• Does not harm accuracy, except in extreme cases

Evolution of Forecast Disagreement

• Lahiri, K., and X Shen, 2008, Evolution of Forecast Disagreement in a Bayesian learning Model, Journal of Econometrics, 144, 325-340.

News and Dispersion of Forecast Revisions

• Forecasts converge, but quarterly GDP releases give forecasters an opportunity to put different spins on the figures

Plan of lecture




Blue Chip Financial Forecasts

Method

• Forecasts from US Blue Chip Financial Forecasters TB3, TB30, RGDP, CPI,1983-1997 (+ updating), Horizons 15 mths – 1 mth

• Questionnaire on Forecaster Characteristics

Sent to 80 BCFF participants, Nov 1993 - Jan 199443 useable responses25 with full track record.

Ranking by Forecast Quality

For the 25 forecasters, we compute average (over horizon) ranks for each target variable by

• Bias (actual-forecast, low rank = overprediction)

• Extremism (absolute deviation from consensus forecast, high rank = far from consensus)

• Accuracy (RMSE, low rank = high accuracy)

• Calculate Rank Correlations with Forecaster Characteristics

Individual Characteristics

Please provide the following information about you and your organisation:

Type of Organisation:Location:Highest College Degree (tick): Bachelors Masters PhD Number of years experience in forecastingPercentage of work time spent in forecastingNumber of staff involved in forecasting at your organisation

Individual effects: Rank Correlations

Bias Extremism AccuracyTB3 TB30 RGDP CPI TB3 TB30 RGDP CPI TB3 TB30 RGDP CPI

FBANK -0.33 -0.36 -0.27 -0.04 -0.35 -0.43 -0.16 -0.32 0.12 -0.10 -0.32 -0.12FNYDC 0.13 -0.18 0.15 -0.06 0.20 0.12 -0.07 0.01 0.04 0.02 0.07 -0.04FDEG 0.44 0.29 -0.12 0.05 0.06 -0.11 -0.16 0.21 -0.38 -0.17 0.12 0.05FYRS -0.17 -0.27 0.24 -0.56 0.29 -0.08 0.28 0.15 0.22 0.19 0.19 0.32FPERCENT 0.17 0.02 0.09 -0.08 0.19 0.20 -0.02 0.27 0.08 0.18 0.31 0.12FNOS -0.06 -0.11 -0.18 -0.02 0.21 0.19 -0.31 -0.14 -0.01 0.01 -0.19 -0.22

• Banks made higher, less extreme, more accurate GDP forecasts

• Experienced forecasters made slightly more extreme forecasts of TB3, RGDP, but no convincing evidence of extremism

• Location, education, attention, size of team not significant

Clientele

Please indicate the relative importance of the following groups as users of your forecasts (weights should add to 100):

Traders inside your organisationOther colleagues inside your organisationClients of your organisationGeneral publicOther

Concentration measures

• We have constructed measures of concentration of Clientele, Technique, Theory and Information weights

A forecaster who only served external clients would have a high concentration measure (UCONC)

A forecaster who put equal weight on all types of user would have a low concentration measure

• Hypothesis is that low concentration may reduce extremism and improve accuracy

Strong Clientele Effects

• Forecasters giving weight to traders, internal users, or public, made less extreme and more accurate forecasts

• Forecasters with external clients made more extreme and less accurate forecasts

• Concentration on one type of client also increases extremism


UTRADERS -0.42 -0.42 -0.13 -0.30 -0.29 -0.20 -0.36 -0.17 0.38 0.06 -0.32 -0.12UINTERNAL -0.05 -0.12 -0.33 -0.14 -0.26 -0.52 -0.12 -0.18 -0.34 -0.41 -0.34 -0.13UCLIENTS 0.33 0.36 0.26 0.36 0.45 0.54 0.35 0.21 0.02 0.25 0.48 0.21UPUBLIC -0.12 -0.04 0.11 -0.11 -0.36 -0.29 -0.23 -0.10 -0.18 -0.04 -0.30 -0.28UOTHER 0.20 0.13 0.26 -0.24 0.07 0.07 0.18 0.32 -0.06 0.02 0.27 0.16UCONC 0.21 0.24 0.21 0.36 0.36 0.47 0.38 0.22 0.16 0.26 0.49 0.23

Forecast Techniques

In making forecasts of US interest rates 3 to 6 months ahead, what weight do you assign to the following forecast techniques: (weights should add to 100)

Econometric Models (structural, regression)Time Series Models (Box Jenkins, ARIMA, VAR)Exponential Smoothing methodsTechnical Analysis (Chart Analysis)JudgmentOther

Also give weights for forecasting 1 year ahead and beyond, if different.

Weak Technique effects

• Smoothing methods are associated with higher accuracy, but little used

• Use of Judgement, and Concentration on one technique increased extremism and reduced accuracy of real GDP forecasts


TECHSEC -0.03 -0.02 -0.36 0.09 0.21 0.29 -0.39 0.01 -0.15 0.07 -0.20 0.02TECHSTS -0.26 -0.45 -0.34 -0.29 0.28 0.06 -0.17 -0.14 0.09 0.09 -0.14 0.16TECHSSM -0.02 -0.20 0.14 -0.31 -0.14 0.00 0.09 0.17 -0.31 -0.28 -0.03 0.00TECHSTA -0.25 -0.22 0.23 -0.26 -0.19 -0.15 0.00 0.01 0.08 0.03 -0.18 -0.14TECHSJT 0.23 0.30 0.34 0.20 -0.16 -0.17 0.50 0.16 0.06 -0.01 0.38 0.11TECHSOTH 0.09 0.08 0.10 0.07 -0.21 -0.15 -0.26 -0.26 0.15 -0.25 -0.05 -0.47TECHSCONC 0.28 0.34 0.42 0.27 -0.19 -0.12 0.33 0.24 0.16 0.10 0.41 0.19

Theory

If you use Econometric Models, what weight do you put on the following types of economic theory: (weights should add to 100)

KeynesianMonetaristRational ExpectationsSupply SideBusiness CycleOther (specify)

Some Theory effects

• Keynesians made low forecasts of interest rates• Monetarists made low forecasts of interest rates, inflation, high

forecasts of real growth. Business Cycle theorists had the opposite biases

• RE theorists made consistently high forecasts of real growth• Few differences in forecast accuracy, however


THKEYNES 0.33 0.34 0.26 0.06 -0.01 0.12 0.06 -0.33 -0.13 -0.17 -0.04 -0.46THMONET 0.21 0.33 -0.40 0.46 -0.47 -0.43 -0.07 0.27 -0.33 -0.24 -0.19 -0.01THRATEX -0.23 -0.11 -0.64 -0.10 0.13 0.07 0.21 0.28 -0.05 0.14 0.00 0.24THSUPSIDE -0.24 -0.06 -0.12 0.12 0.23 -0.04 -0.03 -0.04 0.05 -0.01 0.09 0.07THBUSCYC -0.12 -0.34 0.46 -0.36 0.15 0.18 0.05 -0.08 0.26 0.16 0.14 0.22THOTH 0.02 0.05 -0.10 0.25 -0.19 -0.14 -0.48 0.11 0.02 0.07 -0.14 -0.16THCONC -0.25 -0.31 0.41 -0.26 -0.22 -0.02 -0.01 -0.10 0.44 0.22 0.09 -0.01

Judgment

If you use Judgment, what weight so you place on the following processes? (weights should add to 100)

Own analysis of current eventGroup analysis within your organisation (meetings, surveys)Other (specify)

Some Judgment Process effects

• Some evidence that group processes improve accuracy of interest rate forecasts, reliance on own judgment harms accuracy


JTOWN 0.21 0.47 0.30 0.23 0.00 0.19 0.01 0.26 0.25 0.22 0.16 0.02JTGROUP -0.08 -0.36 -0.30 -0.12 -0.17 -0.27 -0.03 -0.20 -0.38 -0.39 -0.13 -0.07JTOTHER -0.33 -0.41 -0.15 -0.30 0.28 0.03 0.02 -0.22 0.09 0.18 -0.12 0.07

Information

If you use Judgment, what weight so you place on the following pieces of information? (weights should add to 100)

Current Official Economic Statistics (GDP, Inflation, …)Forecasts made by other organisations (e.g. Blue Chip Financial

Forecasts)Surveys of Consumer and Business ConfidenceOther (specify)

Strong Information Source effects

• Forecasters who place a lot of weight on the forecasts of other forecasters are less extreme, and significantly more accurate on interest rate and real GDP forecasts

• Forecasters who rely heavily on one source of information tend to be less accurate


INFONEWS 0.06 0.14 0.01 0.28 0.11 0.23 -0.04 0.04 0.19 0.30 0.15 -0.10INFOFORCS 0.06 -0.11 -0.28 -0.24 -0.26 -0.16 -0.10 -0.01 -0.45 -0.36 -0.51 -0.07INFOSURV 0.13 0.11 0.08 0.15 0.09 -0.13 0.07 0.02 -0.08 -0.15 -0.14 -0.14INFOOTH -0.14 -0.09 0.12 -0.15 0.02 -0.03 0.06 -0.03 0.15 0.03 0.25 0.18INFOCONC -0.04 0.09 -0.03 0.16 0.24 0.27 0.00 0.07 0.28 0.36 0.33 0.12

Who should you listen to?• Less extreme

work in bank serve general public

• More accurate

serve general public use any theory … except RE pay attention to other

forecasters

• More extreme

rational expectations theory listen to friends, boss overweight news

• Less accurate

limited range of users (e.g. only external clients)

doctrinaire use of theory (e.g. only Monetarism)

Why Forecasts Differ and Why are they so Bad? Roy Batchelor Professor of Banking and Finance, Cass...

Documents

Transcript of Why Forecasts Differ and Why are they so Bad? Roy Batchelor Professor of Banking and Finance, Cass...