Collective Revelation: A Mechanism for Self-Verified, Weighted, and Truthful Predictions Sharad...

Collective Revelation: A Mechanism for Self-Verified,

Weighted, and Truthful Predictions

Sharad Goel, Daniel M. Reeves, David M. Pennock

Presented by: Nir Shabbat

Outline

• Introduction–Background and Related Work

• The Settings• A Mechanism For Collective Revelation– The Basic Mechanism–A General Technique for Balancing Budgets

• Summary

Introduction• In many cases, a decision maker may seek to elicit

and aggregate the opinions of multiple experts.• Ideally, would like a mechanism that is:

1. Incentive Compatible - Rewards participants to be truthful

2. Information Weighted - Adjusts for the fact that some experts are better informed than others

3. Self-Verifying - Works without the need for objective, “ground truth” observations

4. Budget Balanced - Makes no net transfers to agent

Background and Related Work

• Proper Scoring Rules – Rewards the agent by assessing his forecast against an actual observed outcome.

• Agents are incentivized to be truthful.• For example, using the Brier scoring rule:

We get:


• Peer Prediction Methods – Can be Incentive Compatible and Self-Verifying.

• For example, the Bayesian truth serum asks agents to report both their own prediction and their prediction of other agents’ predictions.

• Relies on the general property of information- scores, that a truthful answer constitutes the best guess about the most “surprisingly common” answer.


• With the appropriate reward structure , this framework leads to truthful equilibria:

• iff response of agent is • - prediction of player for the frequency of • - geometric average of predictions (or opinions)

and meta predictions


• Predictions Markets – encourage truthful behavior and automatically aggregate predictions from agents with diverse information.

• For example, a prediction market for U.S. presidential elections. Agents buy and sell assets tied to an eventual Democratic or Republican win. Each share pay $1 if the corresponding event occurs.


• Delphi Method – generates consensus predictions from experts generally through a process of structured discussion.

• In each round, each expert anonymously provides his forecast and reasons, at the end of the round, market maker summarize the forecasts and reasons.

• Process is repeated until forecasts converge.


• Competitive Forecasting – elicits confidence intervals on predictions, thereby facilitating information weighting. Like prediction markets, competitive forecasting rewards accuracy, though is not rigorously incentive compatible and relies on benchmarking against objective measurements.

Background and Related WorkIncentive Compatible

Information Weighted

Self-Verifying

Proper Scoring Rules ●

Prediction Markets ○ ●

Peer Prediction ○ ●

Delphi Method ●

Competitive Forecasting ○ ●

Polls ●

Collective Revelation ○ ● ●

The Settings

1. Common prior – Agents & Market Maker have a common prior on the distribution of

2. Independent Private Evidence – Each agent privately observes independent realization of the random variable (independent both of each other and across agents).

3. Rationality – Agents update their beliefs via Bayes’ rule.

4. Risk neutrality – Agents act to maximize their expected payoff.

A Mechanism For Collective Revelation

• Agents are scored against one another’s private information (Self Verification).

• Agents report their subjective expectation of , and their revised estimate in light of new hypothetical evidence (HFS).

• From these 2 reports the subjective posterior, private information and also this agent’s prediction of other agents’ prediction is constructed.

A Mechanism For Collective Revelation

The Basic Mechanism

LEMMA 1

1. An agent has subjective distribution for

It is possible to elicit the kth moment of via a proper scoring rule that pays based on the outcome iff

The Basic Mechanism

• Simply put, scoring against a single Bernoulli observation can only reveal the agent’s expectation and not the uncertainty of it’s prediction (as quantified by the variance).

• So in order for a mechanism to be Information-Weighted (for Bernoulli outcome) it must score agents against multiple observations.

The Basic Mechanism• For example, say we want to elicit an agent’s

prediction regarding the distribution of .• We want to reward the agent using a proper

scoring rule based on a single outcome :

• To elicit the confidence, we must use another outcome . For example, like this:

The Basic Mechanism

The Beta distribution is the “conjugate prior” of the Bernoulli distribution.That is, given with a prior the posterior distribution of , given independent trials out of which are successful, is:

The Basic MechanismLEMMA 2• Let • a prior on • Fix (trials) and (successes) integers• Define:

That is:

The Basic Mechanism

LEMMA 2 (Cont’d)Then:• is a bijection from to

• And

The Basic Mechanism

LEMMA 3• Let • Let a prior on • Let • Define:

The Basic Mechanism

LEMMA 3 (Cont’d)Then:• is a bijection from to

• And:

The Basic Mechanism

THEOREM 1• Consider a Bayesian game according to the

settings shown before with players, .• Let a common prior on • Let be the posterior dist. of agent for after

updating according to its private info.• Fix and integers

The Basic Mechanism

THEOREM 1 (Cont’d)• Suppose that each agent plays action where

The Basic Mechanism

THEOREM 1 (Cont’d)• Define:

Where is projection into the kth component.

The Basic Mechanism

THEOREM 1 (Cont’d)• For arbitrary constants , set the reward of

agent to be:

The Basic Mechanism

THEOREM 1 (Cont’d)• Then:(*)

is a strict Nash Equilibrium.

The Basic Mechanism

PROOF• Fix attention on agent , and suppose all agents

play according to (*).• By Lemma 2 - parameters of .

The Basic Mechanism

PROOF (Cont’d)• Let be private observations of agent . Then: Consequently:

The Basic Mechanism

PROOF (Cont’d)• In particular, and • If plays according to (*) then:

Since we are using the Brier Proper scoring rule, strategy (*) maximizes ’s expected reward.

The Basic Mechanism

PROOF (Cont’d)• Moreover, since is an injection, this is ’s

unique best response, and so strategy (*) is a strict Nash equilibrium.

The Basic Mechanism

CORROLARY• Suppose all agents play the equilibrium

strategy.• Define:

The Basic Mechanism

CORROLARY (Cont’d)• Define the aggregate information-weighted

prediction as:

Let be the posterior dist. Resulting from the cumulative private evidence of all agents.Then .

A General Technique for Balancing Budgets

• The idea is to reward agents via a Shared scoring rule.

A simple application would be:

And this is clearly budget balanced since


• In our setting, the “observations” are determined precisely by agents’ reports – simple application won’t work!

• The solution proposed is to decouple scoring from benchmarking, e.g. agent is rewarded according to its performance relative to a set of agents whose score cannot be affected by agent ’s action.


DEFINITION• Let be a Bayesian game with players• A -projective family of is a family of games

such that for each with , is a Bayesian game restricted to the players that preserves type spaces, action spaces, and players’ beliefs about types. In particular, is determined by a family of reward functions that specifies the expected reward for each player in resulting from any given strategy profile of those players.


THEOREM 2• G is a Bayesian game with n players• is a (strict) Nash equilibrium of G• Suppose is a k-projective family of G such that • Suppose that for each , is a (strict) Nash

equilibrium for .


THEOREM 2 (Cont’d)• Then for any constant , there are player rewards for

the n-player game G such that:1. For any strategy profile s, 2. The original equilibrium q is still a (strict) Nash

equilibrium for the modified gameIn particular, by setting , one can alter the rewards so that the game G is strongly budget balanced.


PROOF• For , denote:– - reward function of player in – - reward function of player in the game – - reward of player in game , when is restricted to

game .


PROOF (Cont’d)• Denote the player sets where the players

wrap around for .• Consider the modified rewards for :


PROOF (Cont’d)• Summing over all players we get:


PROOF (Cont’d)• Further more:

And since , we have , and so for .


PROOF (Cont’d)• Consequently, if is the strategy of player and

is the strategy profile of all other players, then:

Where is a function that does not depend on .


PROOF (Cont’d)• Since is a (strict) Nash equilibrium for :

And so, using the new reward functions, the original equilibrium is still a (strict) Nash equilibrium.

Summary

• We’ve seen a mechanism, i.e. Collective Revelation, for aggregating experts opinions which is:– Incentive Compatible– Information Weighed– Self-Verifying

• We’ve seen a general technique for constructing budget balanced mechanisms that applies both to collective revelation and to past peer-predictions method.

THE END

Collective Revelation: A Mechanism for Self-Verified, Weighted, and Truthful Predictions Sharad...

Documents

Transcript of Collective Revelation: A Mechanism for Self-Verified, Weighted, and Truthful Predictions Sharad...