1

TurKit: A Toolkit for Human Computation Algorithms

Rob Miller & Greg LittleUser Interface Design Group

MIT CSAIL

Joint work with Lydia Chilton, Max Goldman, Jeff Bigham, Aubrey Tatarowicz, Rajeev Najak, Michael Bernstein

June 10, 2010

Outline

• Iterative human computation– TurKit: a toolkit for human

computation algorithms

• Systems with human computation inside– VizWiz: vision for blind users with camera phones– Soylent: putting a crowd inside Microsoft Word

June 10, 2010 2

What is TurKit?

2 Ways

What is TurKit?

Way 1

Way 1

ADD 1, 2

Way 1

ADD 1, 2

JMP somewhere

Way 1

ADD 1, 2

JMP somewhere…

Way 1

ADD 1, 2

JMP somewhere…

TURK “How do you feel?”

Way 1

ADD 1, 2

JMP somewhere…

TURK “How do you feel?”

Way 2

Way 2

Way 2

Way 2

Way 2

Way 2

Way 2

Way 2

Way 2

Way 2

Demo

Demo

TurKit

What can you do with TurKit?

x50

31

19

Improve

Improve

Improve

Vote

Improve

Vote

Handwriting Recognition

Image Description

Progression of Other Offer Description:• $10,000 more• better package• far more competitive offer• more competitive offer• competing offer

Outline to Prose

Picture Sorting

x

Subjective Ratings

Subjective Ratings

Subjective Ratings

Workflow Comparisons

Workflow Comparisons

Workflow Comparisons

Workflow Comparisons

Workflow Comparisons

Iterative

Iterative

Parallel

Iterative

Parallel

Iterative

Parallel

Which is better?

Which is better?

• 30 images

Which is better?

• 30 images• 6 iterations

Which is better?

• 30 images• 6 iterations

Which is better?

Which is better?

Which is better?

Larger Algorithms

Larger Algorithms

Larger Algorithms

Outline

• Iterative human computation– TurKit: a toolkit for human

computation algorithms

• Systems with human computation inside– VizWiz: vision for blind users with camera phones– Soylent: putting a crowd inside Microsoft Word

June 10, 2010 58

VizWiz: Helping the Blind See

59

Which door is the women’s restroom?

joint work with Jeff Bigham (University of Rochester)

June 10, 2010

Helping the Blind See

60

MechanicalTurk

1. the left one2. LEFT3. on the left

June 10, 2010

Field Study

• Field deployment with 11 blind iPhone users– Answers received within a minute for ~5 cents a question– Latency can be reduced to less than 30 seconds by keeping

workers warmed up (at $4 per hour)

June 10, 2010 61

What color is this pillow? What denomination is this bill?

Do you see picnic tables across the parking lot?

What temperature is my oven set to?

Can you please tell me what this can is?

What kind of drink does this can hold?

(89s) I can’t tell.(105s) multiple shades of soft green, blue and gold

(24s) 20(29s) 20

(13s) no(46s) no

(69s) it looks like 425 degrees but the image is difficult to see.(84s) 400(122s) 450

(183s) chickpeas.(514s) beans(552s) Goya Beans

(91s) Energy(99s) no can in the picture(247s) energy drink

All Mobile Users Are Situationally Disabled

62

When is Barack Obama speaking at MIT?

MechanicalTurk

joint work with Rajeev Najak

June 10, 2010

Soylent: Putting a Crowd inside MS Word

June 10, 2010

joint work with Michael Bernstein

63

Crowd-Driven Proofreading

June 10, 2010 64

Crowd-Driven Shortening

June 10, 2010 65

Bigger Idea: REAL Wizard of Oz

• Wizard of Oz is a tried-and-true prototyping technique in AI and HCI– Putting a human behind the curtain until we figure out how to

put software there

• Crowd computing enables Wizard of Oz systems that are useful and deployable– So we can start collecting data about how the system is

really used in practice– Adding the AI backend becomes a performance or cost

optimization

66June 10, 2010

Outline

• Iterative human computation– TurKit: a toolkit for human

computation algorithms

• Systems with human computation inside– VizWiz: vision for blind users with camera phones– Soylent: putting a crowd inside Microsoft Word

• Funded in part by

June 10, 2010 67