Community-based Crowdsourcing

Dipartimento di Elettronica, Informazione eBioingegneria

COMMUNITY-BASED CROWDSOURCING

Marco Brambilla, Stefano Ceri, Andrea Mauri, Riccardo Volonterio

Dipartimento di Elettronica, Informazione e Bioingegneria

3Crowd-based Applications

Emerging crowd-based applications:– opinion mining – localized information gathering – marketing campaigns – expert response gathering

General structure: – the requestor poses some questions – a wide set of responders are in charge of providing answers

(typically unknown to the requestor)– the system organizes a response collection campaign

SOCM’14, Monday, April 7

Com

munit

y-base

d C

row

dso

urc

ing


4Our approach

Combines a conceptual framework, a specification paradigm and a reactive execution control environment

Supports designing, deploying, and monitoring applications on top of crowd-based systems– Design is top-down, platform-independent– Deployment turns declarative specifications into platform-

specific implementations which include social networks and crowdsourcing platforms

– Monitoring provides reactive control, which guarantees applications’ adaptation and interoperability


Com

munit

y-base

d C

row

dso

urc

ing


5Control

Controlling crowdsourcing tasks is a fundamental issue– Cost– Time– Quality

In our approach– Implemented trough active rules

• Control is easily expressible Simple control data structures Familiar formalism

• Support to arbitrarily complex controls Extensibility mechanisms

• Active rules can be system-generated Well-defined semantic


Com

munit

y-base

d C

row

dso

urc

ing


6Community

Set of people that share

Interests

Feature

..or belong to

common entity

social network


Com

munit

y-base

d C

row

dso

urc

ing


7Leveraging communities

Why?– Experts– More engaged

How?– Determine the communities of performers– Monitor them taking into the account the behavior of their

member• Community Control


Com

munit

y-base

d C

row

dso

urc

ing


8Community Control

Community control allow the adaptation of the crowdsourcing campaign

Task / Object allocation

Static / Dynamic

Implemented with the reactive environment present in our approach


Com

munit

y-base

d C

row

dso

urc

ing


10Example (dynamic control)


Com

munit

y-base

d C

row

dso

urc

ing

e: AFTER UPDATE FOR μTObjExecution

c: CommunityControl[CommunityID== NEW.CommunityID].score<=0.5

CommunityControl[CommunityID== NEW.CommunityID].eval=10

a: SET CommunityControl[CommunityID == DB-Group].Enabled = true

?


11Crowdsearcher

A prototype that allows the definition, execution and control of a crowdsourcing campaign according to our approach


Com

munit

y-base

d C

row

dso

urc

ing

http://crowdsearcher.search-computing.org/


12Experiment

16 professors within two research groups in our department (DB and AI groups)

The top 50 images returned by the Google Image API for each query

Each microtask consisted of evaluating 5 images regarding a professor.

Results are accepted (and thus the corresponding object is closed) when enough agreement on the class of the image is reached

Closed objects are removed from new executions.


Com

munit

y-base

d C

row

dso

urc

ing


13Communities

The communities:

the research group of the professor,

the research area containing the group (e.g. Computer Science)

and the whole department (which accounts for more than 600 people in different areas)

Invitations are sent:

inside-out: we started with invitations to experts, e.g. people the same groups as the professor (DB and AI), and then expanded invitations to Computer Science, then to the whole Department, and finally to open social networks (Alumni and PhDs communities on Facebook and Linkedin);

outside-in: we proceeded in the opposite way, starting with the Department members, then restricting to Computer Scientists, and finally to the group's members.


Com

munit

y-base

d C

row

dso

urc

ing


14Number of performers per community


Com

munit

y-base

d C

row

dso

urc

ing

46%

24%

16%

9 / “a lot”


15Precision of performers per community


Com

munit

y-base

d C

row

dso

urc

ing


16Precision of the evaluated objects

Precision decreases with less expert communities

Inside-out strategy (from expert to generic users) outperforms Outside-in strategy (from generic to expert users)


Com

munit

y-base

d C

row

dso

urc

ing


17General observations

Experts from community feel more engaged with the task

They are more demanding with respect to the quality of the application UI and the evaluated objects

Provide feedbacks on the application, question and the objects evaluated– “How is it possible that this image is related to me?!”


Com

munit

y-base

d C

row

dso

urc

ing


19Thanks for you attention

Any question?


Com

munit

y-base

d C

row

dso

urc

ing

Community-based Crowdsourcing

Software

Transcript of Community-based Crowdsourcing