The PollyVote

Combining forecasts for

U.S. Presidential Elections

Andreas Graefe, Karlsruhe Institute of Technology

J. Scott Armstrong, Wharton School, University of Pennsylvania

Randall Jones, Jr., University of Central Oklahoma

Alfred Cuzán, University of West Florida

The full paper to this talk can be downloaded at: tinyurl.com/combiningelections.

Bucharest Dialogues on Expert Knowledge, Prediction, Forecasting: A Social Sciences PerspectiveNovember 21, 2010

Background on the PollyVote project

The PollyVote project was begun in 2003 to demonstrate the value of forecasting principles by applying them to election forecasting.

The initial focus was on combining forecasts.

Performance of the PollyVote

The PollyVote combined forecasts to obtain highly accurate forecasts of U.S. Presidential Election outcomes:– Prospectively for 2004 and 2008 (MAE: 0.4 percentage points)

– Retrospectively for 1992 to 2000

Across these five elections, the PollyVote was on average more accurate than each of its components: - Polls- Prediction markets- Experts- Statistical models

Polly achieved this without knowing anything about politics.

Power of combining

Question: What is the ratio of students per teacher in primary schools in Romania?

Judge Estimate Error

1 18 .5

2 19 1.5

Typical error of individual estimate 1

Combined estimate 18.5 1

Error reduction through combining 0%

Judge Estimate Error

1 18 .5

2 16 1.5

Typical error of individual estimate 1

Combined estimate 17 0.5

Error reduction through combining 50%

Procedure and conditions for combining forecasts

Procedure: Mechanically combine forecasts equal weights

(unless you have strong evidence for differential weights)

Conditions:1. Several forecasts available2. Uncertainty about which forecasts is most accurate

(although combing is often beneficial even when the best method is known beforehand)

Conditions for when combining is most beneficial:1. Different forecasting methods are available2. Forecasts rely upon different data

Benefits of combining

1. Improves accuracy

2. Avoids large errors

3. Provides an additional assessment of uncertainty

4. Can be used for nearly all forecasting problems.

5. Simple to describe and apply.

Costs of combining

1. Requires expertise with various methods

2. Higher expenses with more methods

7

Prior research

Meta-analysis of 30 studies on combining: 12% error reduction vs. error of typical component.

Recommendation: Combine forecasts from different methods that use different information

[Armstrong, 2001]

However, few studies have focused on the ex ante conditions of when combining is most beneficial.

8

9

Polly’sComponents

Polly‘s components

PollsIEM

predictionmarket

ExpertsQuantitative

models

10

Polly’sComponents

Polls

Problem:

• Polls often unreliable, especiallyearly in campaign

• Large differences in results of individual polls conducted aroundthe same time

Polls

Within component Combining

11

Polly’sComponents

IEMprediction

market

Within component Combining

• Polly’s prediction market: Iowa Electronic Markets (IEM)

• 7-day rolling average of daily marketprices

• Adjust for overreactions of marketsuch as information cascades

IEM prediction market

12

Polly’scomponents Experts

Within component Combining

• Survey of experts

• Assumptions: Experts possess

• Information from polls

• Knowledge about the effect of debates, campaigns, etc.

Experts

13

Polly’scomponents

Quantitativemodels

Within combining Combining

Models focus on 2 to 7 variables, most often

Incumbent‘s popularity

State of economy

Individual accuracy of modelsvaries across elections

Quantitative models

14

Mean error reduction(93 days prior to Election Day,1992 to 2008)

Polly’scomponents

Gains from combining within components

Polls IEM Experts Models

Within components Combining Combining Combining Combining

14% 9% 21%18%

Polly’scomponents

Combining across components

Polls IEM Experts Models

Within components Combining Combining Combining Combining

Across componentsCombining(unweighted

average)

PollyVote-Prediction

Mean error reduction(93 days prior to Election Day,1992 to 2008)

Polly’scomponents

Gains from combining across components

Polls(combined)

IEM(combined)

Experts(combined)

Models(combined)

PollyVote-Prediction

50% 1% 32%43%

Mean error reduction(93 days prior to Election Day,1992 to 2008)

Polly’scomponents

Gains from combining within & across components

TypicalPoll

OriginalIEM

TypicalExperts

TypicalModels

PollyVote-Prediction

58% 10% 58%52%

If combining forecasts is so useful,

why is it seldom used?

18

1. Managers do not believe combining helps

In four experiments with MBAs at INSEAD, most subjects did not realize that the error of the average forecast would be less than the error of the typical forecast.

Most subjects thought that averaging forecasts would yield average performance.

[Larrick & Soll, 2006]

19

2. Some forecasters mistakenly believethey are combining properly

People often use unaided judgment to assign differential weights to individual forecasts.

Informal combining is likely to be harmful as people can select a forecast that suits their biases.

20

3. Managers, forecasters, and researchers are persuaded by complexity

Simple models often predict complex problems better than more complex ones.

[Hogarth, in press]

These findings are difficult to believe. There is a strong belief that complex models are necessary to solve complex problems.

21

4. Forecasters build reputation with extreme forecasts

Forecasters do not want to get lost in the crowd.

More extreme forecasts usually gain more attention and the media is more likely to report them.

[Batchelor, 2007]

5. People mistakenly believe they can identify the most accurate forecast

In a series of experiments, when given two estimates as advice, most people chose one instead of averaging them – and thereby reduced accuracy.

[Soll & Larrick, 2009]

Why doesn’t the PollyVote capture mass media attention?

The PollyVote varies little and, basically, is never wrong. Thus, no entertainment value.

Instead of accuracy, voters want excitement – and hope for their candidate.

24

Accuracy problem is solved for major elections

PollyVote deviation averaged 0.4% for the 2004 and 2008 U.S. presidential elections and substantial improvements are scheduled for 2012.

Polly is available to researchers and practitioners for elections in the U.S., as well as in other countries.

25

Applications of combining

All organizations can benefit from combining.

References

Armstrong, J. S. (2001). Combining forecasts. In: J. S. Armstrong (Ed.), Principles of Forecasting: A Handbook for Researchers and Practitioners, Norwell: Kluwer, pp.417-439.

Batchelor, R. (2007). Bias in macroeconomic forecasts, International Journal of Forecasting, 23, 189-203.

Hogarth, R. (in press). When simple is hard to accept. In P. M. Todd, G. Gigerenzer, & The ABC Research Group (Eds.), Ecological rationality: Intelligence in the world. Oxford: Oxford University Press.

Larrick, R. P. & Soll, J. B. (2006). Intuitions about combining opinions: Misappreciation of the averaging principle. Management Science,52, 111-127.

Soll, J. B. & Larrick, R. P. (2009). Strategies for revising judgment: How (and how well) people use others’ opinions, Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 780-805.