HoudiniOn Demand Labor API
By: Chris Conley and Jameson Detweiler
Philly AWS December Meetup
12/13/2010
Easy Mechanical Turk Integration with Houdini
Date: December 13, 2010
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with Houdini
• Marketplace for work
• ~ 500k workers
• Accessible via API
Philly AWS December Meetup
Mechanical Turk
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Task Examples
• Moderation - images, comments, reviews, video
• Categorization - products, best image, tagging
• Ads - moderation, keywords
• Audio Transcription / Video Captioning
• Cleanup - proofreading, contact reconciliation, review editing
• PDF/Image/Handwriting to Text
• Research - contact info, document search
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Turk Overview• Workers and Requesters
• HITs
• Questions
• Assignments
• Qualifications
• Notifications
• Blocking Workers
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Worker View
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Creating HITs
• Easy
• Great for testing the waters
• No integration
• More Flexible
• But Complicated
• Programmatic Integration
WYSIWYG API
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
WYSIWYG
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
WYSIWYG
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
API
http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
HIT Questions
• XML
• External Question
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Turk API Libraries
• ruby-aws http://rubyforge.org/projects/ruby-aws/
• rTurk https://github.com/mdp/rturk
• Turkee https://github.com/aantix/turkee
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Questions about Turk?
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Houdini
• Simple
• Qualified results
• Automatic
• Trusted Workers
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Houdini APIs
• Simple
• Exact
• Multi-answer
• Image Moderation
• Ad Moderation
• Data Reconciliation
• Data Extraction
• ...
Generic Optimized
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Simple API
• One answer per task
• Cheap
• Image Moderation, Low profile content
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Simple API Example
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Exact API
• Multiple workers
• Ensures all answers match
• Returns the “final answer”
• If not, escalated to trusted worker group
• More expensive than Simple
• Image Moderation, Data Harvesting, etc
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Exact API Example
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Multi-answer API
• Ask multiple turkers same question
• All answers are returned
• Surveys, feedback, answer qualification on your end
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Image Moderation API
• No need to build form_html
• Just send the image url
• Optionally send custom instructions and possible responses
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Image Moderation Example
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Image Moderation DemoSource: https://github.com/chrisconley/cat_or_not
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Admin Console• Review Work
• Perform “Trusted” tasks
• Worker Insights: Avg Completion Time, # of workers
• Remove bad performing workers from specific task groups
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Getting Started
• Get an API Key - http://houdiniapi.com/sign-up/
• Tunnlr - http://tunnlr.com
• Sandbox - http://houdini-sandbox.heroku.com
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Pricing Example
Monthly Fee Images Per Month
$20 200
$99 1500
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Other Solutions
• Crowdflower
• Microtask
• CrowdCloud
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Resources
Houdini Docs: http://houdiniapi.com/documentation
Mturk Docs: http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html
Tuesday, December 14, 2010
HoudiniOn Demand Labor API
12/13/2010
Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup
Thanks!Chris Conley Jameson Detweiler
[email protected] [email protected]
@chrisconley @jaymstr
Tuesday, December 14, 2010
Top Related