Accuracy of Small-Group Estimation
description
Transcript of Accuracy of Small-Group Estimation
![Page 1: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/1.jpg)
Accuracy of Small-Group Estimation and the Wisdom of Crowds
Jenny ShiMichael D. Lee
Department of Cognitive ScienceUniversity of California, Irvine
![Page 2: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/2.jpg)
Cheese tray knife set
Crafted of sustainable bamboo and sleek stainless steel, these five distinct blades carve up your brie and Roquefort with ease.
Price in Dollars ($1-$50):
Real price: $38
![Page 3: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/3.jpg)
Experimental Stimuli
100 everyday items
• Images, description, prices
• Between $5-$45
• Obtained through shopping websites
Two sets of 50
• Items uniformly distributed by price
• Each item = 1 trial
![Page 4: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/4.jpg)
Wisdom of Crowds
Groups of people can be smarter than the best individuals among them in the right conditions (Surowiecki, 2005)
A crowd can be “wise” when four conditions are met:
Diversity: Each individual has own unique view
Independence: Less relying on others
Decentralization: Draw info from different sources
Aggregation: Turn individual to collective decision
![Page 5: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/5.jpg)
Preliminary Analysis: Individual vs. Crowds
• Examined the mean average deviation of the price estimations of 22 participants.
• Looking at each individual serves a lower bound
• Standard Wisdom of Crowd analysis serves as upper bound.
![Page 6: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/6.jpg)
Current Study
What if there are only small groups available?
• groups of three individuals
• Between subjects design
• Priming, cooperative and competitive settings
Research questions:
• Which of these settings lead to better or worse estimation of the true prices?
• How does the best setting compare to the individuals and standard wisdom of crowd analysis (our preliminary analyses)?
![Page 7: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/7.jpg)
Experimental Conditions
Condition type
1. With two primes (drawn from previous data sets)
2. Cooperate by hearing each other’s bids
3. Cooperate by agreeing on an estimate
4. Compete with each other by playing the Price is Right game
Participants cooperating or competing with each other estimated sequentially and systematically alternated between each trial.
![Page 8: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/8.jpg)
The Price is Right
Rules: To win the game, player must bid closest to the retail price without going over.
Players can bid as high as they want, but they cannot bid the same amount as others or bid less than $1.
![Page 9: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/9.jpg)
Price is Right encourages strategic estimation
Item for bid: Ipod
$150$165 $1
![Page 10: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/10.jpg)
Cognitive model for competition estimate
(Lee & Shi, 2010)
Bottom line: Instead of aggregating the “raw” estimates from participants that competed, we used a cognitive model to infer their latent knowledge.
wx (a,b,c,μ,σ) πc (c | a,b,μ,σ)= w3 (a b,c,μ,σ)p (μ,σ | a,b,c)p (a,b,c | μ,σ ) p (μ,σ)
…blah blah blah.
![Page 11: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/11.jpg)
Results for small group estimates
$9.36 $8.82 $8.79
![Page 12: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/12.jpg)
Competitive Results
Competitive MAD: $8.05
Primed MAD: $9.36
Cooperative Average MAD: $8.82
Cooperative Consensus MAD: $8.79
Competitive estimate was better than both primed and cooperative
![Page 13: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/13.jpg)
Summary of our results
Wisdom of crowds performs best
• Four conditions were present
Competitive outperformed both cooperative and priming.
• Competing participants discouraged to mimic other bids because of winning incentive.
• Participants that cooperate or were primed may be dependent on other participants in the group or additional information given.
![Page 14: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/14.jpg)
Conclusion
Wisdom of crowd analysis is superior to any other aggregation method.
• Resourceful in extracting information from people.
Competition > Cooperation > Individual
• Groups perform better than individuals in estimation tasks.
• Cooperation worse than competition possibly because lack of independence.
Using cognitive models is an efficient way of combining knowledge across individuals.
• Helps us understand both the observed behavior and the reasoning behind it.
![Page 15: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/15.jpg)
Thanks!
![Page 16: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/16.jpg)
References
Lee, M.D., & Shi, J. (2010). The accuracy of small-group estimation and the wisdom of crowds. In R. Catrambone, & S. Ohlsson (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society.
Surowiecki, J. (2004). The wisdom of crowds. New York: Random House.
![Page 17: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/17.jpg)
Extra Slides
![Page 18: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/18.jpg)
Starbucks stainless steel tumbler
Enjoy your favorite Starbucks brew in this 10-oz. stainless steel tumbler bottle with convenient handles.
Price in Dollars ($1-$50):
Real price: $34
![Page 19: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/19.jpg)
Optimal Price is Right Bidding
For just 3 players, bidding between $1-$50
1 10 20 30 40 500
0.2
0.4
0.6
0.8
1 Player 1
1 10 20 30 40 500
0.2
0.4
0.6
0.8
1 Player 3
1 10 20 30 40 500
0.2
0.4
0.6
0.8
1 Player 4
1 10 20 30 40 500
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 Winning Probabilities
![Page 20: Accuracy of Small-Group Estimation](https://reader033.fdocuments.us/reader033/viewer/2022061112/54573d11b1af9f39378b5163/html5/thumbnails/20.jpg)
Toaster Inference
Participants were shown a $28 toaster
• Bid $31, $28, $1
• Mean of data is $20
• Mean of inferred latent pricedistribution is $29
1 10 20 30 40 50Bid
Pro
babi
lity
1 10 20 30 40 50 1 10 20 30 40 50