Human Computation and Crowdsourcing

Uichin LeeMay 8, 2011

Content Networking• Human intelligence:

– Distributed human computation, crowdsourcing

• Mobile device intelligence:– Sensing (camera, GPS)

• Network intelligence:– Internet, mobile networks (w/ advanced services)

• Application intelligence:– Agent, processing, mining

InternetSmart home/office

On the move Applications

Content provider

Fixed access

Content

Networking

Radioaccess

Device

Network

Application

Human

Crowd

Contents

• Overview• Genres of distributed human computation– Games with a purpose, mechanized labor, wisdom of

crowds, crowdsourcing, dual-purpose work, grand search, human-based genetic algorithms, knowledge collection from volunteer contributors

• Dimensions– Motivation, quality, human skill, participation time,

cognitive load• Analyzing Amazon Mechanical Turk Marketplace

Overview• Distributed human computation (DHC) aims at solving rich

computation problems through collaboration between humans and computers– Particularly, in some problem domains where humans could be much

better than machines– Examples: artificial intelligence, natural language processing, and

computer visions • Well artificial intelligence has been trying hard to solve these

problems using machines– But its quality may be not satisfactory..

• DHC offers the possibility of combining humans and computers: faster than individual human efforts, and quality is as good as human efforts (or even better)

Taxonomy of Distributed Human ComputationAlexander J. Quinn, Benjamin B. Bederson, 2009

Overview

• How? The system has global knowledge of the problem and forms small sub-problems that take advantage of humans’ special abilities– Delegating sub-problems to a large number of people

connected via Internet (could be geographically dispersed)• Examples:

– Searching for a person in a large number of satellite photos covering thousands of square miles of ocean (e.g., Jim Gray)

– Image labeling (e.g., ESP game)– Human-computer interaction, cryptograph, business, genetic

algorithms, etc. (and many others!)

DHC Genres

• Games with a purpose• Mechanized labor• Wisdom of crowds• Crowdsourcing• Dual-purpose work• Grand search• Human-based genetic algorithms• Knowledge collection from volunteer contributors• People sensing

Games with a purpose

• Game that requires the player to perform some computation to gain points or to succeed

• Defining factor: people are motivated by the fun of a game

Mechanized labor

• Crowdsourcing with monetary rewards• Amazon’s Mechanical Turk, ChaCha (paid per micro task)

– Cf) Mturk was lunched in 2005 by the needs of Amazon; they wanted to eliminate all the duplicate pages as much as possible which couldn’t be done using automated algorithms

Wisdom of crowds

• Crowd intelligence: very difficult when done individually, but very easy when aggregated (asking opinions of crowds)

• Example services: online polling, prediction markets

Crowdsourcing• Coined by Jeff Howe in a Wired magazine article

– Displacement of usual internal labor by soliciting unpaid help from the general public

– Motivated by curiosity or serendipity while browsing the web (e.g., online product reviews)

• Examples: – Question answering services: Naver KiN, Yahoo Answer, Askville, Aardvark– Stardust@Home (finding elusive particles from space images)

Dual-purpose work• Translating a computation into an activity that many people were

already doing frequently

Grand search

• Finding a solution (instead of aggregation)• Examples: finding an image that contains something

(e.g., search for a missing person, or for elusive particles as in Stardust@home)

Human-based genetic algorithms

• Humans contribute solutions to problems and subsequent participants by performing functions such as initialization, mutation, and recombinant crossover

• Defining factor is that solutions consists of a sequence of small parts and that they evolve in a way that is controlled by human evaluation

Knowledge collection from volunteer contributors

• Aims to advance artificial intelligence research by using humans to build large databases of common sense facts– E.g., “people cannot brush their hair with

a table”

• Common methods have been using data mining, e.g., Cyc

• Human-based methods could help, e.g., FACTory, Verbosity, 1001 Paraphrases, etc.

People sensing

• Community awareness (participatory sensing)• Emergency/rescue operations

Safecast.org seeks to aggregate worldwide sensor information

Geiger counter;방사능측정기

Pictures from http://news.cnet.com/japan-radiation-monitoring-goes-crowd-open-source/8301-17938_105-20060639-1.html

Dimensions• Motivation

– Pay (e.g., Mturk), altruism (e.g., Naver KiN, Wikipedia), fun (e.g., games), implicit (e.g., embedded in regular activities)

• Quality– Mechanisms: forced agreement (e.g., games), economic models (when money is

involved), defensive task design, redundancy– Checking: statistical, redundant work, multilevel review, expert review, forced

agreement, automatic check, reputation systems• Aggregation

– Knowledge base, statistical, grand search, unit tasks (ChaCha, Mturk)• Human skill

– Language understanding, vision, communications, reasoning, common knowledge/sense

• Participation time: <2min, 2-10min, >10min• Cognitive load (affecting contributor’s willingness to help)

Analyzing the Amazon Mechanical Turk Marketplace

Panagiotis G. Ipeirotis (NYU)

AMT Screenshot

Screenshot

AMT questions

• Who are the workers that complete these tasks?

• What type of tasks can be completed in the marketplace?

• How much does it cost?• How fast can I get results back?• How big is the AMT market place?

Demographics

• Countries: 46.80% US, India: 34%, Misc: 19.2% (from 66 different countries)

http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html

1

Demographics• Why do you complete tasks in Mechanical Turk? Please check any of the

following that applies:– [1] Fruitful way to spend free time and get some cash (e.g., instead of watching TV)– [2] I find the tasks to be fun– [3] To kill time– [4] For "primary" income purposes (e.g., gas, bills, groceries, credit cards)– [5] For "secondary" income purposes, pocket change (for hobbies, gadgets, going out)– [6] I am currently unemployed, or have only a part time job

1 2 3

Demographics• Why do you complete tasks in Mechanical Turk? Please check any of the

following that applies:– [1] Fruitful way to spend free time and get some cash (e.g., instead of watching TV)– [2] I find the tasks to be fun– [3] To kill time– [4] For "primary" income purposes (e.g., gas, bills, groceries, credit cards)– [5] For "secondary" income purposes, pocket change (for hobbies, gadgets, going

out)– [6] I am currently unemployed, or have only a part time job

4 5 6

Type of tasks

Requester distribution

Price distribution

Keywords vs. Ranks

Posting vs. completion rate

Completion time