An Experimental Test of Information & Decision Markets

Robin Hanson, Takashi Ishikida and John Ledyard

Caltech

2/4/2005

25 minute presentation!

25 minute presentation!

DIMACS 2

Information Markets• Standard Information Markets seem to work.

– Small but complete set of securities– Many informationally small and unbiased traders.

• Theory and evidence from experiments and applications are all positive.

• But all assume, require, use, …..– Straight-forward behavior

• Price taking, honest revelation, etc.

– Complete set of state dependent contracts – Common knowledge of all priors, …

• Even then we see “failures to fully aggregate information”– Incomplete Bayesian updating– Incompletely revealing Rational Expectations Equilibrium

DIMACS 3

Decision vs Prediction

• A policy maker does not just want to know the probability that “terrorist attacks in the US will increase in 2005.”

• They want to know the probability that “terrorist attacks in the US will increase in 2005” if “US troops remain in Iraq for 2005.”

• With N events (attacks, troop size, …) and S outcomes for each (increase from 10-20%, decrease, …), a complete set of state dependent contracts requires N^S - 1 contracts.

S = 2, N = 8 => 255 contracts

DIMACS 4

Remember PAM?

Goal: Collect accurate predictive information on political and economic stability in the middle east.

DIMACS 5

Every nation*quarter:-Political stability-Military activity-Economic growth-US $ aid-US military activity& all combinations& ………8 nations, 5 indices,4 quarters

Remember PAM?

DIMACS 6

Every nation*quarter:-Political stability-Military activity-Economic growth-US $ aid-US military activity& all combinations& ………8 nations, 5 indices,4 quarters (N = 180)

•Even if we only use up-down questions, completeness requires 2^180 =1.5*(10^54) contracts.

Remember PAM?

DIMACS 7

Using Conditional Contracts

• The good news– There may more overall trading.

• Traders may know more about and be more willing to trade on the relatively more precise event {terrorism up | troops up} as opposed to the less precise {terrorism up}.

• The bad news– There may be thinner trading per security.

• Too many markets to pay attention to.

– Thinner trading => bad price discovery and incomplete arbitrage => prices do not aggregate information.

DIMACS 8

Decision Markets

• Markets for Decision Analysis will be thin.– Large and possibly incomplete set of securities– Few informationally large and biased traders

• Theory is unlikely to be a good predictor of behavior.

• Current applications and experiments may not be applicable to the thinner situation.

• How can we know what will actually work?

DIMACS 9

Experimental Test Beds

• Create an environment that”captures” as much of the problem as possible (the econ wind tunnel)– Three traders, three events with 2 outcome each (8 states)

– Common prior with asymmetric information• 10 draws from one urn of 6 equally likely => (1,0,1), (1,1,0),…..

• Each trader sees only two entries of each draw: (1,0,x), (1,1,x),…

• Run different mechanisms and market designs

• Measure performance – How close are final prices to the fully informed posterior?

DIMACS 10

3 Variable CDFs

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

KL Distance from Group Posterior

CDF

Prior Distribution

Individual Posterior

Uniform Disribution

Theory Benchmarks - 3 eventsposteriors

priors uniform

DIMACS 11

3 Variable CDFs

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.00E+00 5.00E-02 1.00E-01 1.50E-01 2.00E-01 2.50E-01 3.00E-01 3.50E-01 4.00E-01

KL Distance from Group Posterior

CDF

Individual (72)

Prior Distribution

Individual Posterior

Uniform Disribution

Individual Scoring Ruleposteriors

priors uniform

DIMACS 12

3 Variable CDFs

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

KL Distance from Group Posterior

CDF

Independent DA (24)

Individual (72)

Prior Distribution

Individual Posterior

Uniform Disribution

Standard Marketsposteriors

priors uniform

DIMACS 13

Design Matters• Asking is not enough.• “Let there be markets” is not enough.• Conjecture: An IM will work better in thin

situations, if we use (to “thicken” trading)– Conditional contracts and – a Combinatoric (package bid) Call Market

• Includes “no arbitrage” pricing but is intermittent• Does not directly address “monopolistic agents”

DIMACS 14

3 Variable CDFs

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

KL Distance from Group Posterior

CDF

Independent DA (24)

Combined Value (22)

Individual (72)

Prior Distribution

Individual Posterior

Uniform Disribution

Combinatoric Call Marketposteriors

priors

uniform

DIMACS 15

Design Matters• We are not yet at complete aggregation.• Conjecture: An IM will work even better in

thin situations, if we use (to “thicken” trading)– Conditional contracts and – A Combinatoric Sequentially Shared (Market)

Scoring Rule• Is continuous and directly addresses report

manipulation• But it involves a subsidy to traders.

DIMACS 16

3 Variable CDFs

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

KL Distance from Group Posterior

CDF

Independent DA (24)

Combined Value (22)

Market Maker (8)

Individual (72)

Prior Distribution

Individual Posterior

Uniform Disribution

Shared Scoring Rule - w/CCposteriors

priors

uniform

DIMACS 17

Tentative Conclusions

• Standard markets and surveys do not work will in thin situations.

• Using conditional contracts and assuming some self - selection, either combinatoric call markets or combinatoric sequentially shared scoring rules significantly improve performance over standard markets.

DIMACS 18

Open Questions

• There are many others we did not test– Pari-mutuel mechanisms

• Economides, Lange, and Longitude (some combinatorics)

• Pennock - Dynamic Pari-mutel Market

• Plott - Auction then Pari-mutuel

DIMACS 19

Open Questions

• There are many others we did not test– Pari-mutuel mechanisms

• Economides, Lange, and Longitude (some combinatorics)

• Pennock - Dynamic Pari-mutel Market• Plott - Auction then Pari-mutuel

– Others• HP - • ……….

DIMACS 20

Some Open Questions

• There are many other mechanisms we did not test.– Pari-mutuel mechanisms

• Economides, Lange, and Longitude (some combinatorics)• Pennock - Dynamic Pari-mutel Market• Plott - Auction then Pari-mutuel

– Others• HP - • ……….

• There are many other environments we did not test in.– Information monopolist– External incentives to manipulate internally– And for PAM -- do these results survive in an ultra-thin world?

DIMACS 21

A Force 12 Storm

• Create an environment that really stress-tests the mechanisms– Six traders, 8 events w/ two outcomes each

(256 states)– Common prior with asymmetric information

• 10 draws from one urn of 8! equally likely:– (1,0,1,0,1,1,0,0), (1,1,0,0,0,0,0,0),…..

• Each trader sees only 4 different entries: – (1,0,x,0,x,1,x,x), (1,0,x,0,x,1,x,x), …

DIMACS 22

8 Variable CDFs

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

KL Distance from Group Posterior

CDF

Independent DA (18)

Combined Value (12)

Market Maker (17)

Individual (144)

Prior Distribution

Individual Posterior

Uniform Distribution

posteriors

uniform

priors

DIMACS 23

Summary of Testing• Thin: 3 traders, 3 events

• 7 independent prices from 3 people in 12 minutes• Markets < Individual Scoring Rule < Call < SSSR• SSSR ~ Call given that the group beats the prior

– With selection, SSSR and Call Market do best.

• Ultra-Thin: 6 traders, 8 events • 255 independent prices from 6 people in 12 min. • Markets ~ Individual Scoring Rule ~ Call < SSSR

– SSSR beats the priors at the top (60%)– Nothing else even beats the priors

– SSSR is only one with any aggregation

DIMACS 24

Final Thoughts

• Information Markets are possible and desirable.– Can improve our ability to identify and deal with uncertainty.

• Many policy applications will be in thin situations. • Traditional market designs do not work in thin

situations. – Information monopolies, adverse decisions, partial updating

• The SSSR (w/conditionals) definitely sharpens the signal/noise ratio in thin and ultra-thin markets over traditional markets.

• Can we do better? Undoubtedly.

DIMACS 25