Post on 01-Apr-2015
Why Forecasts Differand Why are they so Bad?
Roy BatchelorProfessor of Banking and Finance, Cass Business School
But back in the real world…
• Forecasters often look (and feel) stupid
• This is becauseSometimes they can’t help it. Sometimes they deliberately make biased forecasts.
• Aim is to separate these causes of error, using evidence from panels of forecastersConsensus EconomicsBlue Chip Financial Forecasts
• Payoff – which forecasters are worth listening to, and when?
Plan of lecture
• Insights from recession forecasts
• Reasons for biased forecasts
• Who is most biased?
Forecasting the 1991 recession
-3.0
-2.0
-1.0
0.0
1.0
2.0
3.0
4.0
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1991 1991 1991 1991 1991 1991 1991 1991 1991 1991 1991 1991
Consensus
Mr. Brightside 1
Dismal Scientist 1
HMT
-5
-4
-3
-2
-1
0
1
2
3
4
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2008 2008 2008 2008 2008 2008 2008 2008 2008 2008 2008 2008 2009 2009 2009 2009 2009 2009 2009 2009 2009 2009 2009 2009
Consensus
Mr. Brightside 2
Dismal Scientist 2
HMT
Forecasting the 2009 recession
Should we privatise GDP forecasts?
• “Ageing model leads Treasury astray” (Sunday Times 9/2/92)
• “Time to take forecasting away from the Treasury” (Sunday Times, 12/4/92).
• What do you think?
Forecasts for 1990
-1
-0.5
0
0.5
1
1.5
2
2.5
3
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
1989 1989 1989 1989 1989 1989 1989 1989 1989 1989 1989 1989 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990
Consensus
Mr. Brightside 1
Dismal Scientist 1
HMT
Forecasts for 2008
-0.5
0
0.5
1
1.5
2
2.5
3
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2008 2008 2008 2008 2008 2008 2008 2008 2008 2008 2008 2008
Consensus
Mr. Brightside 2
Dismal Scientist 2
HMT
Forecasts for 2010: … it’s too soon to know
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2009 2009 2009 2009 2009 2009 2009 2009 2009 2009 2009 2009 2010 2010 2010 2010 2010 2010 2010 2010 2010 2010 2010 2010
Consensus
Mr. Brightside 2
Dismal Scientist 2
HMT
Recession forecasts in general
• Loungani, P, 2001, How accurate are private sector forecasts? Cross-country evidence from consensus forecasts of output growth, International Journal of Forecasting, 17, 419-432
Stylised facts about Forecasts
• It’s hard to forecast recession. These are rare events, always with different causes.
• Consensus forecast usually starts close to the average growth rate, and then adjusts
• Accuracy-improving information arrives only about 12-15 months in advance of the end of the target year.
• Individual forecasts are distributed above and below the consensus in a consistent way. There are persistent optimists and pessimists.
Plan of lecture
• Insights from recession forecasts
• Reasons for biased forecasts
• Who is most biased?
Reasons For Bias
• Incompetence (unlikely, really…)
• Bias in the Consensus:
Learning about Structural Breaks
Market Incentives for Bias
• Bias in Individual Forecasts
Market Incentives for Product Differentiation
Learning about Structural Change
• Important
• Many countries (Japan, Germany, Italy, France) experienced slowdown in trend growth
• Optimal forecast will be biased to optimism as forecasters learn about the new trend
• Evidence supports this - forecasts in US, UK less biased
US – Log(GDP) and Forecast (Non-) Bias
Italy – Log(GDP) and Forecast Bias
Market incentives for bias
• Important for Investment Analysts
• Bias to optimism in earnings forecasts can arise from
Selection of firms/ sectors you believe inRelationship buildingTrade generation
• However, regulatory changes (Sarbanes Oxley) can change incentives…
• May also apply to some government forecasts
Financial analysts: Bias to optimism 1990-2003
Financial analysts: Bias to pessimism 2003-
Dispersion v Herding in Individual Forecasts
• There is evidence of herding – excessive convergence on the consensus – for financial analysts
• However, there is evidence of the excessive dispersion of forecasts by economic forecasters, who
underweight information in the consensus forecast (Batchelor and Dua, 1992, J Forecasting )
Maintain consistently optimistic or pessimistic priors from year to year (Batchelor and Dua, 1990, Int J Forecasting; Batchelor, 2007, Int J Forecasting)
Herding
• Financial forecasters have incentives to overweight consensus forecasts:
“Information cascade”: forecasts are made sequentially, so each forecast becomes part of the next forecasters prediction set. In aggregate published forecasts are biased towards the early forecasts.
“Incentive concavity”: rewards for an accurate “bold” forecast are smaller than penalties for an inaccurate bold forecast. Less experienced forecasters herd more since career prospects are at stake.
Dispersion and Product Differentiation
• Hotelling “location model” – don’t set up your store right next to everyone else, but don’t be too far out of town. So profit maximising strategy is to shade forecasts consistently away from the Consensus (even if you believe it is the best forecast)
• Batchelor and Dua (1990), Batchelor (2007) find individual forecasters persistently make optimistic or pessimistic forecasts relative to the consensus. Interpreted as an attempt to differentiate their product, increase press coverage, book sales, speaking fees etc.
• Does not harm accuracy, except in extreme cases
Evolution of Forecast Disagreement
• Lahiri, K., and X Shen, 2008, Evolution of Forecast Disagreement in a Bayesian learning Model, Journal of Econometrics, 144, 325-340.
News and Dispersion of Forecast Revisions
• Forecasts converge, but quarterly GDP releases give forecasters an opportunity to put different spins on the figures
Plan of lecture
• Insights from recession forecasts
• Reasons for biased forecasts
• Who is most biased?
Blue Chip Financial Forecasts
Method
• Forecasts from US Blue Chip Financial Forecasters TB3, TB30, RGDP, CPI,1983-1997 (+ updating), Horizons 15 mths – 1 mth
• Questionnaire on Forecaster Characteristics
Sent to 80 BCFF participants, Nov 1993 - Jan 199443 useable responses25 with full track record.
Ranking by Forecast Quality
For the 25 forecasters, we compute average (over horizon) ranks for each target variable by
• Bias (actual-forecast, low rank = overprediction)
• Extremism (absolute deviation from consensus forecast, high rank = far from consensus)
• Accuracy (RMSE, low rank = high accuracy)
• Calculate Rank Correlations with Forecaster Characteristics
Individual Characteristics
Please provide the following information about you and your organisation:
Type of Organisation:Location:Highest College Degree (tick): Bachelors Masters PhD Number of years experience in forecastingPercentage of work time spent in forecastingNumber of staff involved in forecasting at your organisation
Individual effects: Rank Correlations
Bias Extremism AccuracyTB3 TB30 RGDP CPI TB3 TB30 RGDP CPI TB3 TB30 RGDP CPI
FBANK -0.33 -0.36 -0.27 -0.04 -0.35 -0.43 -0.16 -0.32 0.12 -0.10 -0.32 -0.12FNYDC 0.13 -0.18 0.15 -0.06 0.20 0.12 -0.07 0.01 0.04 0.02 0.07 -0.04FDEG 0.44 0.29 -0.12 0.05 0.06 -0.11 -0.16 0.21 -0.38 -0.17 0.12 0.05FYRS -0.17 -0.27 0.24 -0.56 0.29 -0.08 0.28 0.15 0.22 0.19 0.19 0.32FPERCENT 0.17 0.02 0.09 -0.08 0.19 0.20 -0.02 0.27 0.08 0.18 0.31 0.12FNOS -0.06 -0.11 -0.18 -0.02 0.21 0.19 -0.31 -0.14 -0.01 0.01 -0.19 -0.22
• Banks made higher, less extreme, more accurate GDP forecasts
• Experienced forecasters made slightly more extreme forecasts of TB3, RGDP, but no convincing evidence of extremism
• Location, education, attention, size of team not significant
Clientele
Please indicate the relative importance of the following groups as users of your forecasts (weights should add to 100):
Traders inside your organisationOther colleagues inside your organisationClients of your organisationGeneral publicOther
Concentration measures
• We have constructed measures of concentration of Clientele, Technique, Theory and Information weights
A forecaster who only served external clients would have a high concentration measure (UCONC)
A forecaster who put equal weight on all types of user would have a low concentration measure
• Hypothesis is that low concentration may reduce extremism and improve accuracy
Strong Clientele Effects
• Forecasters giving weight to traders, internal users, or public, made less extreme and more accurate forecasts
• Forecasters with external clients made more extreme and less accurate forecasts
• Concentration on one type of client also increases extremism
Bias Extremism AccuracyTB3 TB30 RGDP CPI TB3 TB30 RGDP CPI TB3 TB30 RGDP CPI
UTRADERS -0.42 -0.42 -0.13 -0.30 -0.29 -0.20 -0.36 -0.17 0.38 0.06 -0.32 -0.12UINTERNAL -0.05 -0.12 -0.33 -0.14 -0.26 -0.52 -0.12 -0.18 -0.34 -0.41 -0.34 -0.13UCLIENTS 0.33 0.36 0.26 0.36 0.45 0.54 0.35 0.21 0.02 0.25 0.48 0.21UPUBLIC -0.12 -0.04 0.11 -0.11 -0.36 -0.29 -0.23 -0.10 -0.18 -0.04 -0.30 -0.28UOTHER 0.20 0.13 0.26 -0.24 0.07 0.07 0.18 0.32 -0.06 0.02 0.27 0.16UCONC 0.21 0.24 0.21 0.36 0.36 0.47 0.38 0.22 0.16 0.26 0.49 0.23
Forecast Techniques
In making forecasts of US interest rates 3 to 6 months ahead, what weight do you assign to the following forecast techniques: (weights should add to 100)
Econometric Models (structural, regression)Time Series Models (Box Jenkins, ARIMA, VAR)Exponential Smoothing methodsTechnical Analysis (Chart Analysis)JudgmentOther
Also give weights for forecasting 1 year ahead and beyond, if different.
Weak Technique effects
• Smoothing methods are associated with higher accuracy, but little used
• Use of Judgement, and Concentration on one technique increased extremism and reduced accuracy of real GDP forecasts
Bias Extremism AccuracyTB3 TB30 RGDP CPI TB3 TB30 RGDP CPI TB3 TB30 RGDP CPI
TECHSEC -0.03 -0.02 -0.36 0.09 0.21 0.29 -0.39 0.01 -0.15 0.07 -0.20 0.02TECHSTS -0.26 -0.45 -0.34 -0.29 0.28 0.06 -0.17 -0.14 0.09 0.09 -0.14 0.16TECHSSM -0.02 -0.20 0.14 -0.31 -0.14 0.00 0.09 0.17 -0.31 -0.28 -0.03 0.00TECHSTA -0.25 -0.22 0.23 -0.26 -0.19 -0.15 0.00 0.01 0.08 0.03 -0.18 -0.14TECHSJT 0.23 0.30 0.34 0.20 -0.16 -0.17 0.50 0.16 0.06 -0.01 0.38 0.11TECHSOTH 0.09 0.08 0.10 0.07 -0.21 -0.15 -0.26 -0.26 0.15 -0.25 -0.05 -0.47TECHSCONC 0.28 0.34 0.42 0.27 -0.19 -0.12 0.33 0.24 0.16 0.10 0.41 0.19
Theory
If you use Econometric Models, what weight do you put on the following types of economic theory: (weights should add to 100)
KeynesianMonetaristRational ExpectationsSupply SideBusiness CycleOther (specify)
Some Theory effects
• Keynesians made low forecasts of interest rates• Monetarists made low forecasts of interest rates, inflation, high
forecasts of real growth. Business Cycle theorists had the opposite biases
• RE theorists made consistently high forecasts of real growth• Few differences in forecast accuracy, however
Bias Extremism AccuracyTB3 TB30 RGDP CPI TB3 TB30 RGDP CPI TB3 TB30 RGDP CPI
THKEYNES 0.33 0.34 0.26 0.06 -0.01 0.12 0.06 -0.33 -0.13 -0.17 -0.04 -0.46THMONET 0.21 0.33 -0.40 0.46 -0.47 -0.43 -0.07 0.27 -0.33 -0.24 -0.19 -0.01THRATEX -0.23 -0.11 -0.64 -0.10 0.13 0.07 0.21 0.28 -0.05 0.14 0.00 0.24THSUPSIDE -0.24 -0.06 -0.12 0.12 0.23 -0.04 -0.03 -0.04 0.05 -0.01 0.09 0.07THBUSCYC -0.12 -0.34 0.46 -0.36 0.15 0.18 0.05 -0.08 0.26 0.16 0.14 0.22THOTH 0.02 0.05 -0.10 0.25 -0.19 -0.14 -0.48 0.11 0.02 0.07 -0.14 -0.16THCONC -0.25 -0.31 0.41 -0.26 -0.22 -0.02 -0.01 -0.10 0.44 0.22 0.09 -0.01
Judgment
If you use Judgment, what weight so you place on the following processes? (weights should add to 100)
Own analysis of current eventGroup analysis within your organisation (meetings, surveys)Other (specify)
Some Judgment Process effects
• Some evidence that group processes improve accuracy of interest rate forecasts, reliance on own judgment harms accuracy
Bias Extremism AccuracyTB3 TB30 RGDP CPI TB3 TB30 RGDP CPI TB3 TB30 RGDP CPI
JTOWN 0.21 0.47 0.30 0.23 0.00 0.19 0.01 0.26 0.25 0.22 0.16 0.02JTGROUP -0.08 -0.36 -0.30 -0.12 -0.17 -0.27 -0.03 -0.20 -0.38 -0.39 -0.13 -0.07JTOTHER -0.33 -0.41 -0.15 -0.30 0.28 0.03 0.02 -0.22 0.09 0.18 -0.12 0.07
Information
If you use Judgment, what weight so you place on the following pieces of information? (weights should add to 100)
Current Official Economic Statistics (GDP, Inflation, …)Forecasts made by other organisations (e.g. Blue Chip Financial
Forecasts)Surveys of Consumer and Business ConfidenceOther (specify)
Strong Information Source effects
• Forecasters who place a lot of weight on the forecasts of other forecasters are less extreme, and significantly more accurate on interest rate and real GDP forecasts
• Forecasters who rely heavily on one source of information tend to be less accurate
Bias Extremism AccuracyTB3 TB30 RGDP CPI TB3 TB30 RGDP CPI TB3 TB30 RGDP CPI
INFONEWS 0.06 0.14 0.01 0.28 0.11 0.23 -0.04 0.04 0.19 0.30 0.15 -0.10INFOFORCS 0.06 -0.11 -0.28 -0.24 -0.26 -0.16 -0.10 -0.01 -0.45 -0.36 -0.51 -0.07INFOSURV 0.13 0.11 0.08 0.15 0.09 -0.13 0.07 0.02 -0.08 -0.15 -0.14 -0.14INFOOTH -0.14 -0.09 0.12 -0.15 0.02 -0.03 0.06 -0.03 0.15 0.03 0.25 0.18INFOCONC -0.04 0.09 -0.03 0.16 0.24 0.27 0.00 0.07 0.28 0.36 0.33 0.12
Who should you listen to?• Less extreme
work in bank serve general public
• More accurate
serve general public use any theory … except RE pay attention to other
forecasters
• More extreme
rational expectations theory listen to friends, boss overweight news
• Less accurate
limited range of users (e.g. only external clients)
doctrinaire use of theory (e.g. only Monetarism)