Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam,...
-
Upload
emil-bruce -
Category
Documents
-
view
221 -
download
0
Transcript of Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam,...
![Page 1: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/1.jpg)
1
Dynamically Switching between Synergistic Workflows for Crowdsourcing
Christopher H. Lin, Mausam, Daniel S. Weld
University of Washington
AAAI-2012, Toronto, ON, Canada
![Page 2: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/2.jpg)
2
![Page 3: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/3.jpg)
3
Translate AB
![Page 4: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/4.jpg)
4
Translate AB
HumanTrAB
ChooseHumanTr
AB
![Page 5: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/5.jpg)
5
Translate AB
MachineTrAB ProofRd
![Page 6: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/6.jpg)
6
Translate AB
TranslateAC
TranslateCB
![Page 7: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/7.jpg)
7
www.crowdsourcing.com
Is this a face?
Yes No
![Page 8: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/8.jpg)
8
Congressman Tony Coelho landed a million - dollar job on Wall Street after resigning over a controversial junk - bond investment last summer.
NER Tagging
![Page 9: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/9.jpg)
9
Congressman Tony Coelho landed a million - dollar job on Wall Street after resigning over a controversial junk - bond investment last summer.
NER Tagging
location
![Page 10: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/10.jpg)
10
www.crowdsourcing.com
location
film
Congressman Tony Coelho landed a million - dollar job on Wall Street after resigning over a controversial junk - bond investment last summer.
![Page 11: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/11.jpg)
11
www.crowdsourcing.com
Wall Streethttp://en.wikipedia.org/wiki/Wall_StreetWall Street is the financial district of New York City, named after and centered on the eight-block-long street running from Broadway to South Street.…
Wall Street (1987 film)http://en.wikipedia.org/wiki/Wall_Street _(1987_film)Wall Street is a 1987 American drama film released by 20th Century Fox. It was directed by Oliver Stone and stars Michael Douglas, Charlie Sheen, and Daryl Hannah…
Congressman Tony Coelho landed a million - dollar job on Wall Street after resigning over a controversial junk - bond investment last summer.
![Page 12: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/12.jpg)
12
70%
90%
A/B Testing Accuracy
![Page 13: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/13.jpg)
13
70%
90%
A/B Testing Accuracy
![Page 14: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/14.jpg)
14
www.crowdsourcing.com
us_countylocationcitytown
location
Washington StateWashington D.C
![Page 15: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/15.jpg)
15
www.crowdsourcing.com
us_countylocationcitytown
location
Washington StateWashington D.C
![Page 16: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/16.jpg)
16
www.crowdsourcing.com
us_countylocationcitytown
location
Washington StateWashington D.C
![Page 17: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/17.jpg)
17
N Create job using best workflow
[Dai et al, 2010]
![Page 18: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/18.jpg)
18
N Create job using best workflow
[Dai et al, 2010]
SUBOPTIMAL
![Page 19: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/19.jpg)
19
N Create job using best workflow
![Page 20: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/20.jpg)
20
www.crowdsourcing.com
us_countylocationcitytown
location
Washington StateWashington D.C
![Page 21: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/21.jpg)
21
www.crowdsourcing.com
Washingtonhttp://en.wikipedia.org/wiki/WashingtonWashington, D.C., formally the District of Columbia and commonly referred to as Washington, “the District,” or simply D.C., is the capital of the United States….
Washington(state)http://en.wikipedia.org/wiki/Washington_(state)Washington () is a state in the Pacific Northwest region of the United States located north of Oregon, west of Idaho, and south of the Canadian province of British Columbia, on the coast of the Pacific Ocean….
Washington StateWashington D.C
![Page 22: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/22.jpg)
22
www.crowdsourcing.com
Washingtonhttp://en.wikipedia.org/wiki/WashingtonWashington, D.C., formally the District of Columbia and commonly referred to as Washington, “the District,” or simply D.C., is the capital of the United States….
Washington(state)http://en.wikipedia.org/wiki/Washington_(state)Washington () is a state in the Pacific Northwest region of the United States located north of Oregon, west of Idaho, and south of the Canadian province of British Columbia, on the coast of the Pacific Ocean….
Washington StateWashington D.C
![Page 23: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/23.jpg)
23
www.crowdsourcing.com
Washingtonhttp://en.wikipedia.org/wiki/WashingtonWashington, D.C., formally the District of Columbia and commonly referred to as Washington, “the District,” or simply D.C., is the capital of the United States….
Washington(state)http://en.wikipedia.org/wiki/Washington_(state)Washington () is a state in the Pacific Northwest region of the United States located north of Oregon, west of Idaho, and south of the Canadian province of British Columbia, on the coast of the Pacific Ocean….
Washington StateWashington D.C
![Page 24: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/24.jpg)
24
N Create job using best workflow
[Dai et al, 2010]
Before
![Page 25: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/25.jpg)
25
AGENTHUNT
![Page 26: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/26.jpg)
26
AGENTHUNT
Dynamically
![Page 27: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/27.jpg)
27
POMDPStates
Actions
Transition Function
Observation Probabilities
![Page 28: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/28.jpg)
28
POMDPStates
Actions
Transition Function
Observation Probabilities
![Page 29: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/29.jpg)
29
![Page 30: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/30.jpg)
30
2 + 2 = ?
www.crowdsourcing.com
![Page 31: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/31.jpg)
31
2 + 2 = ?
www.crowdsourcing.com
EASY
d = 0
![Page 32: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/32.jpg)
32
www.crowdsourcing.com
?=
![Page 33: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/33.jpg)
33
www.crowdsourcing.com
?=HA
RD
d = 1
![Page 34: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/34.jpg)
34
Correct Answer
Workflow Difficulties
![Page 35: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/35.jpg)
35
POMDPStates
Actions
Transition Function
Observation Probabilities
![Page 36: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/36.jpg)
36
K+1 actions
![Page 37: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/37.jpg)
37
POMDPStates
Actions
Transition Function
Observation Probabilities
![Page 38: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/38.jpg)
38
POMDPStates
Actions
Transition Function
Observation Probabilities
![Page 39: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/39.jpg)
39
![Page 40: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/40.jpg)
40
Difficulty
![Page 41: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/41.jpg)
41
Worker Ability
Difficulty
![Page 42: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/42.jpg)
42Source:Wikipedia
![Page 43: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/43.jpg)
43[Figure courtesy Peng Dai]
4.0
2.0
1.0
0.5
0.25
![Page 44: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/44.jpg)
44
![Page 45: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/45.jpg)
45
LEARN NEW PARAMETERS
![Page 46: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/46.jpg)
46
[Dai et al, 2010]
AgentHunt beats SingleBestWorkflow
![Page 47: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/47.jpg)
47
[Dai et al, 2010]
AgentHunt beats SingleBestWorkflow
![Page 48: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/48.jpg)
48
AgentHunt SingleBestWorkflow
AverageAccuracy (%)
92.45 85.85
AverageNet Utility
-13.36 -18.35
AgentHunt beats SingleBestWorkflow
![Page 49: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/49.jpg)
49
Contributions
• A probabilistic model for multiple workflows
• A decision-theoretic agent that dynamically switches between workflows
• Real experiments that show you should not throw away your workflows
![Page 50: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/50.jpg)
50
INTERESTED?
http://cs.washington.edu/homes/chrislin
![Page 51: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/51.jpg)
51
Repeated Labeling
Romney et al, 1986; Sheng et al, 2008; Raykar et al, 2010; Whitehill et al, 2009; Dai et al, 2010, 2011; Kamar et al, 2012; Parameswaran et al, 2010; Karger et al, 2011; Snow et al, 2008;
Complicated Workflows
Little et al, 2009; Bernstein et al, 2010; Liem et al, 2011; Noronha et al, 2011; Lasecki et al, 2011; Kulkarni et al, 2012;
AI Control of Workflows
Dai et al, 2010,2011; Shahaf et al, 2010; Weld et al, 2011;Kamar et al, 2012;
![Page 52: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/52.jpg)
52
How many people are there in this military base?
www.crowdsourcing.com
Source:Wikipedia
![Page 53: Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto,](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649dc75503460f94abbf96/html5/thumbnails/53.jpg)
53
Christopher H. Lin, Mausam, and Daniel S. Weld. “Crowdsourcing Control: Moving Beyond Multiple Choice.”
In UAI 2012