Post on 15-Jan-2015
description
Best Practices for Sentiment Analysis
Presented by:John HoskinsAmazon Mechanical Turk
Max YankelevichFreedom OSS CrowdControl
© 2011 Amazon.com, Inc. or its Affiliates.
Introductions
© 2011 Amazon.com, Inc. or its Affiliates.
Welcome Logistics
How to ask a question:. Ask a question with the question
panel. We will moderate questions.
Agenda Introductions Sentiment Analysis Applying Mechanical Turk CrowdControl – optimized for Sentiment Q & A
© 2011 Amazon.com, Inc. or its Affiliates.
What is Sentiment Analysis? Keeping your finger on the pulse of your market –
in near real timeWith the explosion in use of Facebook, Twitter, and blogging – it is essential to hear the true Voice of the Customer. Yet keeping up with the round-the-clock information torrent is impossible with old, manual methods. As businesses look to automate the process of filtering out the noise, understanding the conversations, identifying the relevant content and actioning it appropriately, many are now looking to the field of sentiment analysis[
© 2011 Amazon.com, Inc. or its Affiliates.
Why Sentiment Analysis?The ability to lead the competition
in customer satisfaction and support, brand and reputation managementproduct design and marketing
With the proliferation of reviews, ratings, recommendations and other forms of online expression, online opinion has turned into a kind of virtual currency for businesses looking to market their products, identify new opportunities and manage their reputations.
© 2011 Amazon.com, Inc. or its Affiliates.
How to properly analyze Sentiment.
© 2011 Amazon.com, Inc. or its Affiliates.
Context & Cognitive Recognition are the keys to an accurate analysisHuman analysis provides more accurate assessment.Complex emotions such as sarcasm are presented in tweets and blog posts.Wit, sarcasm and complex emotions are difficult to analyze with technology aloneProcessing the unnatural language of text messaging: lol, omgPositive and negative emotions are difficult to accurately assessAvoid misleading classification of SPAM
© 2011 Amazon.com, Inc. or its Affiliates.
Is human judgment affordable and scalable?
Yes.
© 2011 Amazon.com, Inc. or its Affiliates.
Introducing Mechanical TurkAn affordable solution for human judgment.
© 2011 Amazon.com, Inc. or its Affiliates.
Mechanical Turk is a marketplace for work. Mechanical Turk gives businesses and developers access to an on-demand, scalable workforce.
Flexibility: Scale your workforce up and down quickly
Accuracy: Get high quality, efficient and cost effective results.
Price: Pay only when you are satisfied with the results.
Speed: Start receiving results in minutes
© 2011 Amazon.com, Inc. or its Affiliates.
Workforce500,000 Workers190+ Countries
Who are the Workers? Workers are global: 24X7 Follow the Sun
Managing Your Workforce Can narrow to US based (i.e. when you need
western culture competence) Leverage qualifications to find your best Workers
© 2011 Amazon.com, Inc. or its Affiliates.
How it Works
© 2011 Amazon.com, Inc. or its Affiliates.
Popular Use Cases
Data Management Data Verification Data Entry & Collection Data De-duplication Algorithm Training
Content & Media Moderate Photos &
Content Content Creation &
Editing Transcription
Categorization Classification Tagging Sentiment Analysis
Business Services Search Relevancy Product Usability Testing Research
© 2011 Amazon.com, Inc. or its Affiliates.
Case Study – Sentiment Analysis Problem:
A large consumer brand reporting customer need 10’s of thousands of human coded samples to establish a baseline – for each project
Solution: Code them through Mechanical Turk.
Details: Projects are completed in hours providing faster to
market production of broad analysis. By integrating Mechanical Turk into their process, they
freed up analysts time to do value add work and can start projects in a day instead of weeks.
© 2011 Amazon.com, Inc. or its Affiliates.
Strategies for Analyzing Results Plurality
Asking multiple Workers to do assignments (if their answers agree then the result is validated)
Qualifying & Training WorkersAssess competence on your coding instructions before allowing them to work for you
Known Data Sets Include work with known answers to quickly assess worker accuracy.
© 2011 Amazon.com, Inc. or its Affiliates.
Implementing a Sentiment Analysis Workflow
© 2011 Amazon.com, Inc. or its Affiliates.
Freedom at a Glance Long term AWS Partner
Enterprise Cloud Business CrowdControl – fastest growing division
Corporate Office: Newtown PA Regional Offices: Newark, NJ. Reston, VA, Seattle,
WA Offshore Engineering Centers Global operations centers
Privately held Established in 2008 300% annual growth
© 2011 Amazon.com, Inc. or its Affiliates.
Workflow, Adjudication, Worker Management. Programmatic connection to crowdsourcing.How do I know I have the best answer?How do I break my workflow into discrete tasksWho are my best workers?How do I put it all together?
© 2011 Amazon.com, Inc. or its Affiliates.
CrowdControl for Sentiment Analysis Provides High Sentiment Quality Data at Lower
Cost
Ability to Process Large Number of UGC
Combines best of breed Artificial Intelligence to handle most challenging nuances of “Crowdsourcing” for efficient Sentiment Analysis Worker Management Adjudication strategies Workflows Can easily retrieve and send data to and from any data
source (e.g. Database, File ,etc.)
© 2011 Amazon.com, Inc. or its Affiliates.
What Can CrowdControl™ Do for Me?
Mechanical Turk Portal
Turkers
Mechanical Turk Portal
Turkers
Manual S
teps
CustomerIT
Systems
CustomerIT
Systems
Com
ple
te
Auto
matio
n
Quality Information
© 2011 Amazon.com, Inc. or its Affiliates.
Sentiment Analysis Process Setup
© 2011 Amazon.com, Inc. or its Affiliates.
Brand Sentiment Coding Template
© 2011 Amazon.com, Inc. or its Affiliates.
© 2011 Amazon.com, Inc. or its Affiliates.
John Hoskins, Amazon Web Services: hoskins@amazon.comMax Yankelevich, FreedomOSS: myankelevich@freedomoss.com
@amazonmturk
Facebook.com/amazonmturk
Mechanicalturk.typepad.com
© 2011 Amazon.com, Inc. or its Affiliates.
Q & AYour Questions. Answered.