Pattern-Based Specification of Crowdsourcing Applications

31
Pattern-Based Specification of Crowdsourcing Applications Alessandro Bozzon (TU Delft) Marco Brambilla (Politecnico di Milano) Stefano Ceri (Politecnico di Milano) Andrea Mauri (Politecnico di Milano) Riccardo Volonterio (Politecnico di Milano)

description

 

Transcript of Pattern-Based Specification of Crowdsourcing Applications

Pattern-Based Specification of Crowdsourcing Applications

Alessandro Bozzon (TU Delft) Marco Brambilla (Politecnico di Milano)Stefano Ceri (Politecnico di Milano)Andrea Mauri (Politecnico di Milano)Riccardo Volonterio (Politecnico di Milano)

Crowdsourcing and Human Computation

It works like magic!

Endless Applications

Endless Success Stories

2008 Olympics Opening Ceremony

Actually… .

Often a Try&HopeError process

Task Design Matters

Crowd can be unreliable

($) Incentives Matter

Quality Control Matters

Platform of Execution Matters

Setting Nº1 Global Annotations with simple counting

Setting Nº2 Local Annotations with Bounding Boxes

Setting Nº3 Local Annotations with Verified Bounding Boxes

Setting Nº3 Local Annotations with Verified Bounding Boxes

Setting Nº3 Local Annotations with Verified Bounding Boxes

Ok, so what? .

#Workers #Useful Workers #Executions Cost $ Time

(hours) Precision

Setting Nº1 732 44

(6%) 488 72 ~40 ~67%

Setting Nº2 498 25

(5%) 547 48 ~169 ~67%

Setting Nº3 1420 464

(32%) 3387 83 ~184 ~75%

Total 2152 508 4422 203 ~16 days

Our study

Our studyOur Contribution

GOAL !

Simplify and systematize the design, deploy, and monitoring of applications (including experiments)

Contributions

An Abstract Model of

Crowdsourcing Activities

A Composition model for

Crowdsourcing Activities

A Library of crowdsourcing

Patterns

• A conceptual framework • A specification paradigm • A reactive execution control

environment

Models

DEMO VIDEO

Crowd Task

[T operation types](intra-task patterns)

Object Type

block sizemin #obj(cons)

input buffer

batch flow (on closed task)stream flow (on closed object)

MicroTask[MT operation types]

r

data manipulator

Case Study: Movie Scene Analysis

Scenario 1: Scene Positioning

Spoiler Alert!

Order Scenes

Scene in Beg/Mid/End

Scenario 2: Actor Identification

Find Actor

Validate Actor

Position Scenes

[Classify](Static Agreement@3)

MicroTask [Classify]

Scene

block 1min 1

Spoiler Scenes

[Like] (Static Agreement@3)

MicroTask [Like]

Scene

block 1min 1

5

Order Scenes

[Order] (SortByLiking)

MicroTask [Like]

Scene

block 2min 2

[Class=E]

[Class=B OR M]

Example of Scenario 1 Model

PatternsIntra-Task Auxiliary Workflow

Intra-Task

Pre-Processing Post-Processing

Task

Consensus

Splitting

Assignment

AggregationmicroTaskmicroTaskmicroTask

microTaskmicroTaskmicroTaskmicroTaskmicroTaskmicroTask

• Consensus • Join • Sort

• Grouping • Performer Control

• Planning • Assignment • Aggregation • Quality & Performer

Auxiliary Intra-Task

Pre-Processing Post-Processing

Task

Consensus

Splitting

Assignment

AggregationmicroTaskmicroTaskmicroTask

microTaskmicroTaskmicroTaskmicroTaskmicroTaskmicroTask

• Pruning • Tie Breaking

• Operations before or after the execution

Workflow• Set of heterogeneous

tasks

Create Decide

Improve Compare / Verify

Find Fix

(a)

(b)

(c)

Auxiliary Task

Create/Decide

Improve/Compare

Find/Fix/Verify

Experiments

1700 Executions

39$

September 2013

Streaming Vs. Batch Execution

Position Scenes

[Classify] (Static Agreement@3)

MicroTask [Classify]

Scene

block 1min 1

Spoiler Scenes

[Like](Static Agreement@3)

MicroTask [Like]

Scene

block 1min 1

7

Order Scenes

[Order](SortByLiking)

MicroTask [Like]

Scene

block 2min 2

[Class=E]

[Class=B OR M]

(P1)5

3

Position Scenes

[Classify](Static Agreement@3)

MicroTask [Classify]

Scene

block 1min 1

Spoiler Scenes

[Like](Static Agreement@3)

MicroTask [Like]

Scene

block 1min 1

7

Order Scenes

[Order](SortByLiking)

MicroTask [Like]

Scene

block 2min 2Cons.

[Class=E]

[Class=B OR M]

(P2)5

3

P.Beg P.Mid P.EndP1 0.5 1 0.11P2 0.5 0.8 0.33

Spear.Beg Spear. Mid.MidP1 0.5 0.54

P2 0.9 0.51

PositionOrder

P1PositionOrder

P2

b) Elapsed Time (Mins)

#Clo

sed

Obj

ects

11020304050607080

5 60 120 180 240 300 360 5 60 120 180 240 300 360

PositionOrder

P1PositionOrder

P2

b) Elapsed Time (Mins)

#Clo

sed

Obj

ects

11020304050607080

5 60 120 180 240 300 360 5 60 120 180 240 300 360

PositionOrder

P1PositionOrder

P2

b) Elapsed Time (Mins)

#Clo

sed

Obj

ects

11020304050607080

5 60 120 180 240 300 360 5 60 120 180 240 300 360

Intra-Task Consensus Vs. Workflow Decision

A4 A5 A6Precision 0.99 0.95 0.89

Recall 0.90 1 0.96F-Score 0.93 0.97 0.90

Find Actors

[Tag] (Static Agreement@3)

MicroTask [Tag]

Scene

block 1min 1

Validate Actors

[Like]

MicroTask [Like]

Scene+Actor

block Allmin 1

5

(A4)

Find Actors

[Tag]

MicroTask [Tag]

Scene

block 1min 1

Validate Actors

[Like] (Majority Voting@2)

MicroTask [Like]

Scene+Actor

block Allmin 1

5 3

(A5)

(A6)

Find Actors

[Tag](Static Agreement@3)

MicroTask [Tag]

Scene

block 1min 1

Validate Actors

[Like] (Majority Voting@2)

MicroTask [Like]

Scene+Actor

block Allmin 1

5 3

count(Actor.Like)<=1

ActorValidate

A3ActorValidate

A4ActorValidate

A5

a) Elapsed Time (Mins)

ActorValidate

A6

#Clo

sed

Obj

ects

0102030405060

5 30 60 90 120 160

5 30 60 90 120 160 5 60 120 180 240 60 300 540 780

ActorValidate

A3ActorValidate

A4ActorValidate

A5

a) Elapsed Time (Mins)

ActorValidate

A6

#Clo

sed

Obj

ects

0102030405060

5 30 60 90 120 160

5 30 60 90 120 160 5 60 120 180 240 60 300 540 780

ActorValidate

A3ActorValidate

A4ActorValidate

A5

a) Elapsed Time (Mins)

ActorValidate

A6

#Clo

sed

Obj

ects

0102030405060

5 30 60 90 120 160

5 30 60 90 120 160 5 60 120 180 240 60 300 540 780

ActorValidate

A3ActorValidate

A4ActorValidate

A5

a) Elapsed Time (Mins)

ActorValidate

A6

#Clo

sed

Obj

ects

0102030405060

5 30 60 90 120 160

5 30 60 90 120 160 5 60 120 180 240 60 300 540 780

ActorValidate

A3ActorValidate

A4ActorValidate

A5

a) Elapsed Time (Mins)

ActorValidate

A6

#Clo

sed

Obj

ects

0102030405060

5 30 60 90 120 160

5 30 60 90 120 160 5 60 120 180 240 60 300 540 780

ActorValidate

A3ActorValidate

A4ActorValidate

A5

a) Elapsed Time (Mins)

ActorValidate

A6

#Clo

sed

Obj

ects

0102030405060

5 30 60 90 120 160

5 30 60 90 120 160 5 60 120 180 240 60 300 540 780

Take-Home Message

Engineering approaches should be applied to people management too

In crowdsourcing, control and monitoring are key for successful outcomes

Questions?