Pattern-Based Specification of Crowdsourcing Applications

download Pattern-Based Specification of Crowdsourcing Applications

of 31

  • date post

    23-Aug-2014
  • Category

    Science

  • view

    618
  • download

    12

Embed Size (px)

description

 

Transcript of Pattern-Based Specification of Crowdsourcing Applications

  • Pattern-Based Specication of Crowdsourcing Applications Alessandro Bozzon (TU Delft) Marco Brambilla (Politecnico di Milano) Stefano Ceri (Politecnico di Milano) Andrea Mauri (Politecnico di Milano) Riccardo Volonterio (Politecnico di Milano)
  • Crowdsourcing and Human Computation It works like magic! Endless Applications Endless Success Stories 2008 Olympics Opening Ceremony
  • Actually Often a Try&HopeError process Task Design Matters Crowd can be unreliable ($) Incentives Matter Quality Control Matters Platform of Execution Matters
  • Setting N1 Global Annotations with simple counting
  • Setting N2 Local Annotations with Bounding Boxes
  • Setting N3 Local Annotations with Veried Bounding Boxes
  • Setting N3 Local Annotations with Veried Bounding Boxes
  • Setting N3 Local Annotations with Veried Bounding Boxes
  • Ok, so what? #Workers #Useful Workers #Executions Cost $ Time (hours) Precision Setting N1 732 44 (6%) 488 72 ~40 ~67% Setting N2 498 25 (5%) 547 48 ~169 ~67% Setting N3 1420 464 (32%) 3387 83 ~184 ~75% Total 2152 508 4422 203 ~16 days
  • Our study
  • Our study Our Contribution
  • GOAL ! Simplify and systematize the design, deploy, and monitoring of applications (including experiments)
  • Contributions An Abstract Model of Crowdsourcing Activities A Composition model for Crowdsourcing Activities A Library of crowdsourcing Patterns A conceptual framework A specication paradigm A reactive execution control environment
  • Models
  • DEMO VIDEO
  • Crowd Task [T operation types] (intra-task patterns) Object Type block size min #obj (cons) input buffer batch ow (on closed task) stream ow (on closed object) MicroTask [MT operation types] r data manipulator
  • Case Study: Movie Scene Analysis Scenario 1: Scene Positioning Spoiler Alert! Order Scenes Scene in Beg/Mid/End Scenario 2: Actor Identication Find Actor Validate Actor
  • Position Scenes [Classify] (Static Agreement@3) MicroTask [Classify] Scene block 1 min 1 Spoiler Scenes [Like] (Static Agreement@3) MicroTask [Like] Scene block 1 min 1 5 Order Scenes [Order] (SortByLiking) MicroTask [Like] Scene block 2 min 2 [Class=E] [Class=B OR M] Example of Scenario 1 Model
  • Patterns Intra-Task Auxiliary Workow
  • Intra-Task Pre-Processing Post- Processing Task Consensus Splitting Assignment Aggregation microTaskmicroTaskmicroTask microTaskmicroTaskmicroTask microTaskmicroTaskmicroTask Consensus Join Sort Grouping Performer Control Planning Assignment Aggregation Quality & Performer
  • Auxiliary Intra-Task Pre-Processing Post- Processing Task Consensus Splitting Assignment Aggregation microTaskmicroTaskmicroTask microTaskmicroTaskmicroTask microTaskmicroTaskmicroTask Pruning Tie Breaking Operations before or after the execution
  • Workow Set of heterogeneous tasks Create Decide Improve Compare / Verify Find Fix (a) (b) (c) Auxiliary Task Create/Decide Improve/Compare Find/Fix/Verify
  • Experiments 1700 Executions 39$ September 2013
  • Streaming Vs. Batch Execution Position Scenes [Classify] (Static Agreement@3) MicroTask [Classify] Scene block 1 min 1 Spoiler Scenes [Like] (Static Agreement@3) MicroTask [Like] Scene block 1 min 1 7 Order Scenes [Order] (SortByLiking) MicroTask [Like] Scene block 2 min 2 [Class=E] [Class=B OR M] (P1) 5 3 Position Scenes [Classify] (Static Agreement@3) MicroTask [Classify] Scene block 1 min 1 Spoiler Scenes [Like] (Static Agreement@3) MicroTask [Like] Scene block 1 min 1 7 Order Scenes [Order] (SortByLiking) MicroTask [Like] Scene block 2 min 2 Cons. [Class=E] [Class=B OR M] (P2) 5 3 P.Beg P.Mid P.End P1 0.5 1 0.11 P2 0.5 0.8 0.33 Spear.Beg Spear. Mid .MidP1 0.5 0.54 P2 0.9 0.51 Position Order P1 P b) Elapsed Tim #ClosedObjects 1 10 20 30 40 50 60 70 80 5 60 120 180 240 300 360 5 Position Order P1 Position Order P2 b) Elapsed Time (Mins) #ClosedObjects 1 10 20 30 40 50 60 70 80 5 60 120 180 240 300 360 5 60 120 180 240 300 360 Position Order P1 P b) Elapsed Tim #ClosedObjects 1 10 20 30 40 50 60 70 80 5 60 120 180 240 300 360 5
  • Intra-Task Consensus Vs. Workow Decision A4 A5 A6 Precision 0.99 0.95 0.89 Recall 0.90 1 0.96 F-Score 0.93 0.97 0.90 Find Actors [Tag] (Static Agreement@3) MicroTask [Tag] Scene block 1 min 1 Validate Actors [Like] MicroTask [Like] Scene+Actor block All min 1 5 (A4) Find Actors [Tag] MicroTask [Tag] Scene block 1 min 1 Validate Actors [Like] (Majority Voting@2) MicroTask [Like] Scene+Actor block All min 1 5 3 (A5) (A6) Find Actors [Tag] (Static Agreement@3) MicroTask [Tag] Scene block 1 min 1 Validate Actors [Like] (Majority Voting@2) MicroTask [Like] Scene+Actor block All min 1 5 3 count(Actor.Like)