Post on 10-May-2015
description
REACTIVE CROWDSOURCING
Alessandro Bozzonab
Marco Brambillaa
Stefano Ceria
Andrea Mauria
aPolitecnico di Milano
Dipartimento di Elettronica, Informazione e BioIngegneria
bDelft University Of Technology
Department of Software And Computer Technology
2
Crowd Control is tough… • There are several aspects that makes crowd
engineering complicated • Task design, planning, assignment• Workers discovery, assessment, engagement
Wednesday, May 15 Reactive Crowdsourcing
http://xkcd.com/1060/
3
Crowd Control is tough…
Wednesday, May 15 Reactive Crowdsourcing
• There are several aspects that makes crowd engineering complicated • Task design, planning, assignment• Workers discovery, assessment, engagement
• Goal: taming the crowd• Cost• Time• Quality
4
Crowd Control is tough…
Wednesday, May 15 Reactive Crowdsourcing
• There are several aspects that makes crowd engineering complicated • Task design, planning, assignment• Workers discovery, assessment, engagement
• Goal: taming the crowd• Cost• Time• Quality• Motivation!
• Need for higher level abstractions and tools• CONTROL as first-class citizen
5
Reactive Crowdsourcing• A conceptual framework for modeling crowdsourcing
computations and control requirements• Task Design• Reactive Control Design
• Active Rule programming framework• Declarative rule language
• A reactive execution environment for requirement enforcement and reactive execution• Based on the CrowdSearcher approach
Wednesday, May 15 Reactive Crowdsourcing
6
Why Active Rules?• Crowdsourcing control typically focuses on task data
• Execution results, agreement on truth value, workers performance
• An active rule approach can provide• Ease of Use: control is easily expressible
• Simple control data structures • Familiar formalism
• Power: support to arbitrarily complex controls• Extensibility mechanisms
• Automation: most active rules can be system-generated• Well-defined semantic
• Flexibility: simple control variants have localized impact on the rules set• Control isolation
Wednesday, May 15 Reactive Crowdsourcing
7
The CrowdSearcher Approach• Human-Enhanced data management with social networks
and Q&A systems as crowdsourcing platforms
• Example: search task (WWW2012)
Wednesday, May 15 Reactive Crowdsourcing
Human Interaction Management
Social Networks
HumanComputation
PlatformsQ&A
Data Management System
Human Access Interface
Remote Data Access
Local Data Access
Task
Human-EnhancedData
Query
Results
8
• A simple abstract model• A task receives a list of input objects • Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects
• A simple task design and deployment process, based on specific data structures• created using model-driven transformations• driven by the task specification
The Design Process
Wednesday, May 15 Reactive Crowdsourcing
More in the paper
I O
9
• A simple abstract model• A task receives a list of input objects • Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects
• A simple task design and deployment process, based on specific data structures• created using model-driven transformations• driven by the task specification
The Design Process
Task Specification Task Planning Control Specification
Wednesday, May 15 Reactive Crowdsourcing
• Task Spec: task operations, objects, and performers Dimension Tables
10
• A simple abstract model• A task receives a list of input objects • Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects
• A simple task design and deployment process, based on specific data structures• created using model-driven transformations• driven by the task specification
The Design Process
Task Specification Task Planning Control Specification
Wednesday, May 15 Reactive Crowdsourcing
• Task Spec: task operations, objects, and performers Dimension Tables• Task Planning: work distribution Execution Table for task monitoring
11
• A simple abstract model• A task receives a list of input objects • Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects
• A simple task design and deployment process, based on specific data structures• created using model-driven transformations• driven by the task specification
The Design Process
Task Specification Task Planning Control Specification
Wednesday, May 15 Reactive Crowdsourcing
• Task Spec: task operations, objects, and performers Dimension Tables• Task Planning: work distribution Execution Table for task monitoring• Control Specification: task control policies Control Mart
12
Task Specification_1/3
Wednesday, May 15 Reactive Crowdsourcing
• Operation Types: Choice, Like, Score, Tag, Classify, Order, …• Operation Parameters: e.g. classification classes
Task
tIDopType
categories
Task Specification Task Planning Control Specification
Task Configuration
t1Classify
Rep/Dem
13
Task Specification_2/3
Wednesday, May 15 Reactive Crowdsourcing
Politician
classifiedParty
lastNamephoto
oID
• Input Objects Schema: typed attributes• Output Attributes (according to task type)
Task
tIDopType
categories
Task Specification Task Planning Control Specification
Task ConfigurationObject
Specification
o1 Obama http://…. ?????
14
Task Specification_3/3
Wednesday, May 15 Reactive Crowdsourcing
Politician
classifiedParty
lastNamephoto
oID
Task ConfigurationObject
SpecificationPerformer
Specification
• Execution platform(s)• Qualifications, etc.
Task
tIDopType
categories PerformernamepID
platform
Task Specification Task Planning Control Specification
p1 Alessandro Facebook
15
Task Planning_1/2
• Organize the task in MicroTasks, and allocate input objects• μTaskObjectExecution Designed for execution monitoring
• Track performers response
classifiedPartyplatform
μTaskObjectExecution
μtIDstartTs
endTsoIDpID
Wednesday, May 15 Reactive Crowdsourcing
Politician
classifiedParty
lastNamephoto
oID
Task
tIDopType
categories PerformernamepID
platform
Splitting
Task Specification Task Planning Control Specification
mt1 O1 … …
… …
16
Task Planning_2/2
• Assign performers to MicroTasks on platforms• Pull: dynamic assignment (First come - First served / Choice of the
performer)• Push: static assignment (Performers’ priority / Performer matching)
classifiedPartyplatform
μTaskObjectExecution
μtIDstartTs
endTsoIDpID
Wednesday, May 15 Reactive Crowdsourcing
Politician
classifiedParty
lastNamephoto
oID
Task
tIDopType
categories PerformernamepID
platform
Splitting Assignment
Task Specification Task Planning Control Specification
mt1 O1 P1 Republican
00:00:01 00:00:10 Facebook
17
Control Specification_1/4
Wednesday, May 15 Reactive Crowdsourcing
Task Specification Task Planning Control Specification
• Status Variable: tracking task and performers status
classifiedPartyplatform
μTaskObjectExecution
μtIDstartTs
endTsoIDpID
Politician
classifiedParty
lastNamephoto
oID
PerformernamepID
statusplatformTask
tIDopType
categoriesstatus Trusted/SpammerCreated/Planned/Closed
18
Control Specification_2/4
Wednesday, May 15 Reactive Crowdsourcing
Task Specification Task Planning Control Specification
• Object : tracking objects status
classifiedPartyplatform
μTaskObjectExecution
μtIDstartTs
endTsoIDpID
Politician
classifiedParty
lastNamephoto
oID
PerformernamepID
statusplatformTask
tIDopType
categoriesstatus
ObjectControl #dem
oID#eval
#rep
#curAnswer
19
Control Specification_3/4
Wednesday, May 15 Reactive Crowdsourcing
Task Specification Task Planning Control Specification
• Object : tracking object responses• Performer: tracking performer behavior (e.g. spammers)
PerformerControl #right
pID#eval
#wrong
classifiedPartyplatform
μTaskObjectExecution
μtIDstartTs
endTsoIDpID
Politician
classifiedParty
lastNamephoto
oID
PerformernamepID
statusplatformTask
tIDopType
categoriesstatus
ObjectControl #dem
oID#eval
#rep
#curAnswer
20
Control Specification_4/4
• Object : tracking object responses• Performer: tracking performer behavior (e.g. spammers)• Task: tracking task status: closing @completion, re-plan
Wednesday, May 15 Reactive Crowdsourcing
TaskControl#compObj
tID#compExec
PerformerControl #right
pID#eval
#wrong
classifiedPartyplatform
μTaskObjectExecution
μtIDstartTs
endTsoIDpID
Politician
classifiedParty
lastNamephoto
oID
PerformernamepID
statusplatformTask
tIDopType
categoriesstatus
ObjectControl #dem
oID#eval
#rep
#curAnswer
Task Specification Task Planning Control Specification
21
Active Rules Language• Active rules are expressed on the previous data structures• Event-Condition-Action paradigm
Wednesday, May 15 Reactive Crowdsourcing
22
Active Rules Language• Active rules are expressed on the previous data structures• Event-Condition-Action paradigm
• Events: data updates / timer• ROW-level granularity
• OLD before state of a row• NEW after state of a row
Wednesday, May 15 Reactive Crowdsourcing
e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
23
Active Rules Language• Active rules are expressed on the previous data structures• Event-Condition-Action paradigm
• Events: data updates / timer• ROW-level granularity
• OLD before state of a row• NEW after state of a row
• Condition: a predicate that must be satisfied (e.g. conditions on control mart attributes)
Wednesday, May 15 Reactive Crowdsourcing
e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
c: NEW.ClassifiedParty == ’Republican’
24
Active Rules Language• Active rules are expressed on the previous data structures• Event-Condition-Action paradigm
• Events: data updates / timer• ROW-level granularity
• OLD before state of a row• NEW after state of a row
• Condition: a predicate that must be satisfied (e.g. conditions on control mart attributes)
• Actions: updates on data structures (e.g. change attribute value, create new instances), special functions (e.g. replan)
Wednesday, May 15 Reactive Crowdsourcing
e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
c: NEW.ClassifiedParty == ’Republican’
a: SET ObjectControl[oID == NEW.oID].#Eval+= 1
More in the paper
25Wednesday, May 15 Reactive Crowdsourcing
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example
TaskControl#compObj
tID PerformerControl
μTaskObjectExecution
Politician classifiedPartyoID
PerformerTask
ObjectControl #dem
oID
#rep
#evaltIDEvent
26Wednesday, May 15 Reactive Crowdsourcing
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example
TaskControl#compObj
tID PerformerControl
μTaskObjectExecution
Politician classifiedPartyoID
PerformerTask
ObjectControl #dem
oID
#rep
#evaltID
Condition
27Wednesday, May 15 Reactive Crowdsourcing
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example
TaskControl#compObj
tID PerformerControl
μTaskObjectExecution
Politician classifiedPartyoID
PerformerTask
ObjectControl #dem
oID
#rep
#evaltID
Action
28Wednesday, May 15 Reactive Crowdsourcing
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example
TaskControl#compObj
tID PerformerControl
μTaskObjectExecution
Politician classifiedPartyoID
PerformerTask
ObjectControl #dem
oID
#rep
#evaltID
Action
29
Rule Programming Best Practice• We define three classes of rules
Wednesday, May 15 Reactive Crowdsourcing
μTaskObjectExecution
PerformerControl
ObjectControl
TaskControl
Politician Performer Task
30
Rule Programming Best Practice
Wednesday, May 15 Reactive Crowdsourcing
• We define three classes of rules• Control rules: modifying the control tables;
μTaskObjectExecution
PerformerControl
ObjectControl
TaskControl
Politician Performer Task
31
Rule Programming Best Practice
Wednesday, May 15 Reactive Crowdsourcing
• We define three classes of rules• Control rules: modifying the control tables; • Result rules: modifying the dimension tables (object, performer, task);
μTaskObjectExecution
PerformerControl
ObjectControl
TaskControl
Politician Performer Task
32
Rule Programming Best Practice
Wednesday, May 15 Reactive Crowdsourcing
• Top-to-bottom, left-to-right, evaluation• Guaranteed termination
• We define three classes of rules• Control rules: modifying the control tables; • Result rules: modifying the dimension tables (object, performer, task);
μTaskObjectExecution
PerformerControl
ObjectControl
TaskControl
Politician Performer Task
33
Rule Programming Best Practice• We define three classes of rules
• Control rules: modifying the control tables; • Result rules: modifying the dimension tables (object, performer, task); • Execution rules: modifying the execution table, either directly or through re-planning
Wednesday, May 15 Reactive Crowdsourcing
μTaskObjectExecution
PerformerControl
ObjectControl
TaskControl
Politician Performer Task
• Termination must be proven (Rule precedence graph has cycles)
34
Experimental Evaluation
• GOAL: demonstrate the flexibility and expressive power of reactive crowdsourcing
• 3 experiments, focused on Italian politicians• Parties: Human Computation affiliation classification• Law: Game With a Purpose guess the convicted politician• Order: Pure Game hot or not
• 1 week (November 2012)• 284 distinct performers
• Recruited through public mailing lists and social networks announcements
• 3500 Micro Tasks
Wednesday, May 15 Reactive Crowdsourcing
35
Politician Affiliation• Given the picture and name of a politician, specify his/her political
affiliation• No time limit• Performers are encouraged to look up online
• 2 set of rules• Majority Evaluation• Spammer Detection
Wednesday, May 15 Reactive Crowdsourcing
36
Results – Majority Evaluation_1/3
Wednesday, May 15 Reactive Crowdsourcing
30 object; object redundancy = 9;
Final object classification as simple majority after 7 evaluations
37
Results - Majority Evaluation_2/3
Wednesday, May 15 Reactive Crowdsourcing
Final object classification as total majority after 3 evaluations
Otherwise, re-plan of 4 additional evaluations. Then simple majority at 7
38
Results - Majority Evaluation_3/3
Wednesday, May 15 Reactive Crowdsourcing
Final object classification as total majority after 3 evaluations
Otherwise, simple majority at 5 or at 7 (with replan)
39
Results – Spammer Detection_1/2
Wednesday, May 15 Reactive Crowdsourcing
New rule for spammer detection without ground truth
Performer correctness on final majority. Spammer if > 50% wrong classifications
40
A Short CrowdSearcher Demo
Wednesday, May 15 Reactive Crowdsourcing
41
Summary• Results
• An integrated framework for crowdsourcing task design and control• Well-structured control rules with some guarantees of termination• Support for cross-platform crowd interoperability• A working prototype crowdsearcher.search-computing.org
• Forthcoming• Exploitation of interoperability• Expertise finding• Dynamic planning• Integration with other social-networks and human computation
platforms
Wednesday, May 15 Reactive Crowdsourcing
42
QUESTIONS?
Wednesday, May 15 Reactive Crowdsourcing