Best Practices for Sentiment Analysis

8/6/2019 Best Practices for Sentiment Analysis

1/25

B e s t P r a c t i c e sf o r

S e n t i m e n tA n a l y s i s

Presented by:John HoskinsAmazon Mechanical Turk

Max YankelevichFreedom OSS CrowdControl


2/25

2011 Amazon.com, Inc. or its Affiliates.

Introductions

John HoskinsAmazon Mechanical Turk

Sr. Manager, Business Development

Max YankelevichFreedomOSSChief Architect


3/25


Welcome Logistics

How to ask a question:.

Ask a question with the questionpanel.

We will moderate questions.

Agenda

Introductions

Sentiment Analysis

Applying Mechanical Turk

CrowdControl optimized for Sentiment

Q & A


4/25


What is Sentiment Analysis? Keeping your finger on the pulse of your market in

near real timeWith the explosion in use of Facebook, Twitter, and blogging it isessential to hear the true Voice of the Customer. Yet keeping up withthe round-the-clock information torrent is impossible with old,manual methods. As businesses look to automate the process offiltering out the noise, understanding the conversations, identifyingthe relevant content and actioning it appropriately, many are nowlooking to the field of sentiment analysis[


5/25


Why Sentiment Analysis?

The ability to lead the competition in customer satisfaction and support,

brand and reputation management

product design and marketing

With the proliferation of reviews, ratings,recommendations and other forms of online expression,

online opinion has turned into a kind of virtual currencyfor businesses looking to market their products, identifynew opportunities and manage their reputations.


6/25


How to properly analyzeSentiment.


7/25 2011 Amazon.com, Inc. or its Affiliates.

Context & Cognitive Recognitionare the keys to an accurate analysisHuman analysis provides more accurate assessment. Complex emotions such as sarcasm are presented in tweets and

blog posts.

Wit, sarcasm and complex emotions are difficult to analyze withtechnology alone

Processing the unnatural language of text messaging: lol, omg

Positive and negative emotions are difficult to accurately assess

Avoid misleading classification of SPAM



Is human judgmentaffordable and scalable?

Yes.



Introducing Mechanical TurkAn affordable solution for human judgment.



Mechanical Turk is amarketplace for work.Mechanical Turk gives businesses and developers access to

an on-demand, scalable workforce. Flexibility: Scale your workforce up and down quickly

Accuracy: Get high quality, efficient and cost effectiveresults.

Price: Pay only when you are satisfied with the results. Speed: Start receiving results in minutes



Workforce

500,000 Workers

190+ Countries

Who are the Workers? Workers are global: 24X7 Follow the Sun

Managing Your Workforce Can narrow to US based (i.e. when you need western

culture competence)

Leverage qualifications to find your best Workers



How it Works

Design Your Task

(HIT)

Publish your HIT tothe marketplace

You TheRequester

Accepts your HIT

Submits an answer

The Worker Validate the Results

Pay Workers

Go!

You TheRequester



Popular Use Cases

Data Management Data Verification

Data Entry & Collection

Data De-duplication

Algorithm Training

Content & Media Moderate Photos & Content

Content Creation & Editing

Transcription

Categorization Classification

Tagging

Sentiment Analysis

Business Services Search Relevancy

Product Usability Testing

Research



Case Study Sentiment Analysis Problem:

A large consumer brand reporting customer need 10s ofthousands of human coded samples to establish a baseline for each project

Solution:Code them through Mechanical Turk.

Details:

Projects are completed in hours providing faster to marketproduction of broad analysis.

By integrating Mechanical Turk into their process, they freed upanalysts time to do value add work and can start projects in a dayinstead of weeks.



Strategies for Analyzing Results

PluralityAsking multiple Workers to do assignments (iftheir answers agree then the result is validated)

Qualifying & Training WorkersAssess competence on your coding instructionsbefore allowing them to work for you

Known Data Sets

Include work with known answers to quicklyassess worker accuracy.



Implementing a SentimentAnalysis Workflow



Freedom at a Glance Long term AWS Partner

Enterprise Cloud Business

CrowdControl fastest growing division

Corporate Office: Newtown PA Regional Offices: Newark, NJ. Reston, VA, Seattle, WA

Offshore Engineering Centers

Global operations centers

Privately held

Established in 2008

300% annual growth



Workflow, Adjudication,Worker Management.Programmatic connection to crowdsourcing. How do I know I have the best answer?

How do I break my workflow into discrete tasks

Who are my best workers?

How do I put it all together?



CrowdControl for Sentiment Analysis Provides High Sentiment Quality Data at Lower Cost

Ability to Process Large Number of UGC

Combines best of breed Artificial Intelligence to handle mostchallenging nuances of Crowdsourcing for efficientSentiment Analysis

Worker Management

Adjudication strategies Workflows

Can easily retrieve and send data to and from any datasource (e.g. Database, File ,etc.)



What Can CrowdControl Do for Me?

Mechanical Turk Portal

Turkers

Mechanical Turk Portal

Turkers

ManualSteps

CustomerIT

SystemsStep

Create sentiment coding templates

Step Come up with opinion adjudication rules

Step Extract data from Internal and External IT Systems

Step Review responses

Step Ban bad workers

Step Accept or reject answers

Step Pay workers

Step Assign special worker qualifications

Step

Import data back into IT Systems

CustomerIT

Systems

Complete

Automation

QualityInformation



Sentiment Analysis Process Setup



Brand Sentiment Coding Template



John Hoskins, Amazon Web Services:

[email protected]

Max Yankelevich, FreedomOSS:

[email protected]

@amazonmturk

Facebook.com/amazonmturk

Mechanicalturk.typepad.com


25/25

Q & AYour Questions. Answered.

Best Practices for Sentiment Analysis

Documents

Transcript of Best Practices for Sentiment Analysis