Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... ·...
Transcript of Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... ·...
![Page 1: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/1.jpg)
Reward
Determination in
Crowdsourcing
Kiatten MittonsNathan DeMaria | Rees Klintworth | Derek Nordgren
![Page 2: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/2.jpg)
Citation
Amos Azaria, Yonatan Aumann, and Sarit Kraus (2014). Automated agents for
reward determination for human work in crowdsourcing applications,
autonomous agents and multiagent systems, 28(6):934-955.
![Page 3: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/3.jpg)
Problem
Introduction
Rees Klintworth
![Page 4: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/4.jpg)
Research Goal
Automate the assignment of differing individual
rewards in a crowdsourcing application
![Page 5: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/5.jpg)
Crowdsourcing
● Goal broken into tiny increments of worko Too many to manually determine reward for each
● For simplicity, most problems are divided
with equal rewardso Doesn’t make economic sense
![Page 6: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/6.jpg)
Problem Setting
● Small, identical tasks must be completed
● Any one of available human workers can
complete a task
● There are costs associated with bringing in
new worker and making an offer
![Page 7: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/7.jpg)
TCP
● TCP - Task Completion Problemo A = set of tasks
o O = set of possible rewards (payment) for a task
o T = set of types
Decision functions for workers
● TCP(A, pi) - minimize cost to satisfy all tasks
in A, subject to type distribution pi
![Page 8: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/8.jpg)
M-TCP
● Each task is broken up into milestones
● Each milestone for a task must be
completed by the same worker
● A worker can leave mid-tasko Compensated for milestones completed
● Upfront or Stepwise
![Page 9: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/9.jpg)
Requester
● Makes an offer to a worker for a tasko If declined, can either:
Move on to a new worker
Make another offer to the worker
● Has a history of offers and their
acceptance/rejection
● Must assign each task to some worker
![Page 10: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/10.jpg)
Workers
● Infinite amount of workers exist
● Any worker can complete any task
● Each worker has a decision policy governed
by historyo Rejects only
● Any cost for the worker is included in
decision policy
![Page 11: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/11.jpg)
Minimizing Cost
● The requester attempts to minimize the total
cost associated with completing all tasks
● Doesn’t care about:o Best candidate
o Cheapest candidate for a particular task
![Page 12: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/12.jpg)
Problem is NP-Hard
● Maps to Set-Cover problem
● Set-Cover is a known NP-Hard problem
● Solution set grows exponentially large
![Page 13: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/13.jpg)
Simplifying Assumptions
● Two restrictions considered in order to solve
the problem
● RPBA - each worker type has a reservation
price
● NBA - only one offer is made to each worker
● Different algorithm for each type
![Page 14: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/14.jpg)
Bargaining Effect
● Basis for NBA
● If given a higher offer after declining a lower
offer, you are less likely to accept the higher
offer than if only offered the higher offer
● Explored in referenced work
![Page 15: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/15.jpg)
Agents
Nathan DeMaria
![Page 16: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/16.jpg)
Reservation Price-Based Agent (RPBA)
● Type of RPBA agents is defined by their
reservation price
● Accept any offer above
● Reject any offer below
● Requester knows:o all reserve prices
o frequencies of each reservation price
![Page 17: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/17.jpg)
Optimal Algorithm for RPBA
● Offers must be exactly reservation prices
● For each reservation priceo Estimate the cost of an interaction with that agent
assuming you will negotiate up to that price, and
then move on to the next agent if that maximum
price is declined
● Find the reservation price that minimizes that
cost, use the table for that price
![Page 18: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/18.jpg)
RPBA Algorithm for Stepwise M-TCP
● Similar to basic RPBA, with an added layer
of complexityo Offers need to be negotiated at each milestone
o Requester knows frequencies of each type (a type is a
unique combination of reservation prices for each
milestone)
o Based on milestone offers accepted/rejected so far, the
requestor adjusts the probabilities that the agent is of
each type
![Page 19: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/19.jpg)
No-Bargaining Agent (NBA)
● Assumes workers will consider first offer
● Workers require a much higher price for
second offer
![Page 20: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/20.jpg)
Optimal Algorithm for NBA
u(x) = probability of worker accepting $x offer
Expected cost per worker
Expected cost per assignment
![Page 21: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/21.jpg)
Type Elicitation - RPBA
● Vickrey Auctiono For crowdsourcing, use the last 3 bids
● Becker-DeGroot-Marschak mechanismo User submits a bid
o Random number is generated
o User is paid the generated number if it is greater
than the bid
● Both incentivize truthful bidding
● Cluster bids into reservation prices
![Page 22: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/22.jpg)
Type Elicitation - NBA
● Sigmoid function is assumed
● Pick test offers
● Observe % accepted
at each offer
● Find u(x) by fitting a
sigmoid function to
those points
![Page 23: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/23.jpg)
Experimentation
Derek Nordgren
![Page 24: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/24.jpg)
Experimentation Setup
● Performed using Amazon’s Mechanical Turk
service
● Both single milestone and 5-milestone tasks
were simulated
● New worker cost = $0.20
● New offer cost = $0.04
![Page 25: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/25.jpg)
Experimentation Setup
● Re-performed experiment from Related
Work
● Performed new experiment
● RPBA, NBA and human “experts”
performance were analyzed
![Page 26: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/26.jpg)
Experiment 1
● Previously published experiment
● Workers are required to identify a unique
shape
![Page 27: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/27.jpg)
Experiment 2
● Workers are required to encode a paragraph
of text
● TCP - Requester must accomplish 25 tasks
● M-TCP - Requester must accomplish 25, 5-
milestone tasks
![Page 28: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/28.jpg)
Results - Aggregated across Experiments
● The NBA performs best, though only slightly
better than RPBA
● Both agents significantly outperform human
“experts”
![Page 29: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/29.jpg)
Results, cont’d
![Page 30: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/30.jpg)
Results, cont’d
![Page 31: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/31.jpg)
Agent Recommendation
The authors recommend the NBA agent
● Easier to implement
● All workers are paid identically (more fair)
● Data collection (sigmoid calibration) is
simpler than RPBA’s auction process
![Page 32: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/32.jpg)
Agent RecommendationRe
The authors recommend the NBA agent
● May significantly outperform the RPBA in
some conditions
● Every interaction builds a more accurate
strategy
![Page 33: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/33.jpg)
Merits of Bargaining
In general, bargaining was found to be
“fruitless.”
● Adds to expense
● Does not improve completion rate
NBA avoids bargaining altogether.
![Page 34: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/34.jpg)
Sunk Cost Effect
NBA also avoids “sunk cost effect”
● In M-TCPs, human’s cost was much higher
● People tend to spend more money when
money has already been spent
● Requires more research
![Page 35: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/35.jpg)
Reward Determination Schedule
Both agents performed much better with
upfront reward determination schedule.
● Reduced costs ~ 15%
● Requester commitment lowers cost required
by workers
● Human agents incurred more costs upfront
![Page 36: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/36.jpg)
Cost Modification
● A higher Cc results in higher offers
● A lower Co means more offers from RPBA
● Changes in Co do not affect NBA
![Page 37: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/37.jpg)
Future Work
Incorporate milestone repetitiveness into NBA
to account for worker
● Expertise
● Boredom
![Page 38: Reward Determination in Crowdsourcingcse.unl.edu/~lksoh/Classes/CSCE475H_Spring15/seminars/... · 2015. 4. 9. · NBA - only one offer is made to each worker Different algorithm for](https://reader036.fdocuments.us/reader036/viewer/2022071406/60fd669149b16c18c1514912/html5/thumbnails/38.jpg)
Questions?