CrowdFlower University Oct. 21 2013
61
CrowdFlower University October 21, 2013
-
date post
14-Sep-2014 -
Category
Education
-
view
922 -
download
2
description
On Monday, Oct. 21, CrowdFlower ran its first session of CrowdFlower University. The turn out was incredible. We had more than 30 people representing a range of organizations, from startups to Fortune 100 companies. It was an engaging 4 hour session filled with great questions.
Transcript of CrowdFlower University Oct. 21 2013
CrowdFlower University October 21, 2013
1:00 - 1:05 // Arrive, Plug-in
1:05 - 1:30 // Intro to Workforce Management and the CrowdFlower Platform
1:30 - 2:30 // Process Deconstruction & Job Creation
2:30 - 2:40 // Break
2:40 - 3:30 // Job Monitoring and Reports
3:30 - 4:00 // From the Field: Case Studies
4:00 - 4:10 // Break
4:10 - 4:30 // What’s to come: Product Roadmap
4:30 - 5:00 // Q&A
Schedule for the day
2 CrowdFlower, Inc. – Proprietary and Confidential
CrowdFlower University will teach you the CrowdFlower
Platform, empowering you to manage workforces online.
Purpose
3 CrowdFlower, Inc. – Proprietary and Confidential
4 CrowdFlower, Inc. – Proprietary and Confidential
Success Center
Success Center
5 CrowdFlower, Inc. – Proprietary and Confidential
success.crowdflower.com
6 CrowdFlower, Inc. – Proprietary and Confidential
Workforce Management
Geography
Performance Levels and Skills
Workflows
Test Questions
Channels
Workforce Management
7 CrowdFlower, Inc. – Proprietary and Confidential
8 CrowdFlower, Inc. – Proprietary and Confidential
Geography
9 CrowdFlower, Inc. – Proprietary and Confidential
Geography
10 CrowdFlower, Inc. – Proprietary and Confidential
Performance Levels and Skills
Skills and Performance Levels
11 CrowdFlower, Inc. – Proprietary and Confidential
The CrowdFlower Platform enables you to add
Skill requirements and Performance Levels to
your job. Enabling the Skilled Crowds will likely
increase the quality of your results and may
decrease the throughput of your judgments.
12 CrowdFlower, Inc. – Proprietary and Confidential
Workflows
13 CrowdFlower, Inc. – Proprietary and Confidential
Workflows 6RXUFH�
�,QSXW�3D\ORDG�
&RPSUHVV�WR���URZ�SHU�FRPSDQ\
9HULI\��)LQG�2IILFLDO�85/V)LQG�3DJH�1XPEHUV
)LQG�2IILFHU��'LUHFWRU�3DJHV
$GG�1HZ
([SDQG�WR���URZ�SHU�OHDGHU
/RFDWH�/HDGHUV�RQ�3DJHV��9HULI\�'HOHWH�
)LQG�&XUUHQW�7LWOHV��%LRV
$GG�1HZ
([SDQG�WR���URZ�SHU����OHDGHUV
/RFDWH�/HDGHUV�RQ�3DJHV��9HULI\�'HOHWH�
([SDQG�WR���URZ�SHU�OHDGHU
)LQG�&XUUHQW�7LWOHV��%LRV
)LQG�8QLYHUVLWLHV )LQG�$JH)LQG�*HQGHU
)LQG�0DMRU��'HJUHH�DQG�*UDGXDWLRQ�<HDU
)LQG�8QLYHUVLWLHV)LQG�$JH )LQG�*HQGHU
)LQG�0DMRU��'HJUHH�DQG�*UDGXDWLRQ�<HDU
)LQG�)XOO�1DPH )LQG�)XOO�1DPH
1DPH�3DUVLQJ /515��0DWFK�JHQGHU�EDVHG�RQ�ELR�NH\ZRUGV
/515��0DWFK�JHQGHU�EDVHG�RQ�ELR�NH\ZRUGV
HQG�QRGH
HQG�QRGH
PLVVLQJ�SGI PLVVLQJ�RIILFLDO�XUO
XQYHULILHG
XQYHULILHG
HQG�QRGH
QR�XQLYHUVLWLHV
HQG�QRGH
QR�XQLYHUVLWLHV
HQG�QRGH
QR�ELRV
HQG�QRGH
QR�ELRV
3DFNDJLQJ
3') 85/
)LQG�&HUWLILFDWLRQV )LQG�&HUWLILFDWLRQV
-RE
6FULSW
HQG�QRGH
6RXUFH��,QSXW�3D\ORDG�
.(<
/515
/515�7LWOH�1RUPDOL]DWLRQ
14 CrowdFlower, Inc. – Proprietary and Confidential
Test Questions
Test Questions
15 CrowdFlower, Inc. – Proprietary and Confidential
Test Questions are a powerful method for
measuring the quality of the contributors in your
job. By providing the answers to a small
percentage of units, you can tell how accurate
someone is, and automatically remove anyone
who doesn't meet your requirements.
16 CrowdFlower, Inc. – Proprietary and Confidential
Channels
Channels
17 CrowdFlower, Inc. – Proprietary and Confidential
CrowdFlower Terminology
18 CrowdFlower, Inc. – Proprietary and Confidential
• Judgment - One (of many) answers given by an individual contributor
• Unit - One row on a spreadsheet (One company, one officer)
• Dataset - The entire set of data for a project
• Task - A subset of data from the whole dataset
• Workflow - Multiple jobs that strung together
• Channel - A website in which a contributor can access a task
• Test Questions - Training data created in the platform
• Contributor - Person completing assignments online
CrowdFlower Terminology
19 CrowdFlower, Inc. – Proprietary and Confidential
Process Deconstruction and Job Creation
20 CrowdFlower, Inc. – Proprietary and Confidential
Let’s learn how to build a job!
Today we’re going to build and run a sentiment analysis job about CrowdFlower.
21 CrowdFlower, Inc. – Proprietary and Confidential
Process Deconstruction & Job Creation
22 CrowdFlower, Inc. – Proprietary and Confidential
Best Practices – Job Construction
• Break down problem into smallest components
• Leverage the Template Library
• Use the Instructions Template
• Help contributors find the data (provide links, images, etc)
• Provide examples in the instructions
• Leverage CML logic wherever possible
• Use validations to control text inputs
• Allow an "escape" option for multiple choice questions regarding information in the job or a link: "I don't know", "I couldn't find it", "the link is broken", etc.
Best Practices – Job Construction
23 CrowdFlower, Inc. – Proprietary and Confidential
24 CrowdFlower, Inc. – Proprietary and Confidential
Best Practices – Test Questions
• Try to create an even distribution of answers
• Enable random data for Test Questions
• Copy units to source
• Be as thorough in your Test Question reasons as possible – these will help train the crowd throughout the task
• Create Test Questions out of edge case (subjective or tricky) units to train the crowd
• You don’t need to answer every question
• For subjective questions you can allow for multiple correct answers
25 CrowdFlower, Inc. – Proprietary and Confidential
Best Practices – Test Questions
26 CrowdFlower, Inc. – Proprietary and Confidential
Best Practices – Job Calibration
• Set the Max Judgments Per IP at the number of units in an assignment multiplied by the amount of gold in the task
• This ensures that a contributor will not see the same gold unit twice
• Set variable judgments on trickier CML elements
• Set threshold for trust of contributors
Best Practices – Job Calibration
27 CrowdFlower, Inc. – Proprietary and Confidential
Launch!
28 CrowdFlower, Inc. – Proprietary and Confidential
29 CrowdFlower, Inc. – Proprietary and Confidential
Task Monitoring and Reports
Task Monitoring
Interpreting Reports
Interpreting Results
*This session will teach you how to understand your results.
Task Monitoring and Reports
30 CrowdFlower, Inc. – Proprietary and Confidential
31 CrowdFlower, Inc. – Proprietary and Confidential
Task Monitoring
• Review Test Questions
• Monitor Contributors
Task Monitoring
32 CrowdFlower, Inc. – Proprietary and Confidential
33 CrowdFlower, Inc. – Proprietary and Confidential
Interpreting Reports
Interpreting Reports
34 CrowdFlower, Inc. – Proprietary and Confidential
35 CrowdFlower, Inc. – Proprietary and Confidential
Interpreting Results
• Confidence Scores
• Aggregation Methods
• Auditing for Accuracy
Interpreting Results
36 CrowdFlower, Inc. – Proprietary and Confidential
37 CrowdFlower, Inc. – Proprietary and Confidential
Case Studies
Case Studies
38 CrowdFlower, Inc. – Proprietary and Confidential
In this session we will cover several practical applications
of the CrowdFlower Platform.
39 CrowdFlower, Inc. – Proprietary and Confidential
Search Relevance Testing
40 CrowdFlower, Inc. – Proprietary and Confidential
Example 1
41 CrowdFlower, Inc. – Proprietary and Confidential
Example 2
42 CrowdFlower, Inc. – Proprietary and Confidential
Master Data Management
43 CrowdFlower, Inc. – Proprietary and Confidential
Find URL
44 CrowdFlower, Inc. – Proprietary and Confidential
Find About Us Page
45 CrowdFlower, Inc. – Proprietary and Confidential
Categorize Businesses
46 CrowdFlower, Inc. – Proprietary and Confidential
De-Duplication
47 CrowdFlower, Inc. – Proprietary and Confidential
Content Categorization
48 CrowdFlower, Inc. – Proprietary and Confidential
Photo Categorization
49 CrowdFlower, Inc. – Proprietary and Confidential
Product Categorization
50 CrowdFlower, Inc. – Proprietary and Confidential
Content Moderation
51 CrowdFlower, Inc. – Proprietary and Confidential
Photo Moderation
52 CrowdFlower, Inc. – Proprietary and Confidential
Text Moderation
53 CrowdFlower, Inc. – Proprietary and Confidential
Image and Text Interpretation
Image Transcription
54 CrowdFlower, Inc. – Proprietary and Confidential
OCR Verification
55 CrowdFlower, Inc. – Proprietary and Confidential
56 CrowdFlower, Inc. – Proprietary and Confidential
Product Roadmap
Automatic Test Question access for Enterprise customers
only
A wizard for first-time users so that your team needs less
training
A reputation score for our contributors and improvements
to our performance levels
Q4 2013
57 CrowdFlower, Inc. – Proprietary and Confidential
In-platform templates for you to get started quickly
Improvements to our GUI
Team collaboration on jobs
In-platform auditing tool so that your internal team and
our strongest contributors can easily evaluate the output
of your job
Q1 2014
58 CrowdFlower, Inc. – Proprietary and Confidential
New task analytics for improved slicing and dicing of your
data
Workflow building tool
Minimize human input (e.g. variable pay where you set
the budget, we set the pay for contributors)
And beyond…
59 CrowdFlower, Inc. – Proprietary and Confidential
60 CrowdFlower, Inc. – Proprietary and Confidential
Questions?
Nathan Zukoff
Platform Program Manager
973-997-1921
Contact Me
61 CrowdFlower, Inc. – Proprietary and Confidential
