Designing Ranking Systems for Consumer Reviews: Designing Ranking Systems for Consumer Reviews:

The Economic Impact of Customer Sentiment in Electronic MarketsThe Economic Impact of Customer Sentiment in Electronic Markets

Anindya GhoseAnindya GhosePanagiotis IpeirotisPanagiotis Ipeirotis

Stern School, New York UniversityStern School, New York University

ICEC 2007

Democratization of content creationDemocratization of content creation

Opinions in user-generated content

– The Web has changed the way that people express their views and opinions – One can express opinions on almost anything at:

• review sites, forums, reputation profiles, blogs

Opinions Matter

• Businesses: marketing intelligence, product design and service improvement.

• Firms invest a lot to find consumer opinions• Consultants, Surveys and focus groups, etc

• Individuals: interested in other’s opinions on products, services, topics, events

• Online word-of-mouth

Selected Prior ResearchSelected Prior Research

• Analyze and classify sentiments in online opinions – Pang and Lee (2004), Hu and Liu (2004), Kim and Hovy (2004), Liu, Hu and

Cheng (2005)

• Volume and valence in online reviews influences product sales

– Dellarocas et al. (2005), Chevalier and Mayzlin (2006), Liu (2006), Clemons and Gao (2006)

• Reviewer’s social information influences product sales and review helpfulness over and above volume and valence– Forman, Ghose and Wiesenfeld (2006)

What about the impact of textual sentiments on •Product sales

•Review Helpfulness

Focus of this workFocus of this work

Our QuestionsOur Questions• How sentiment in the text affects How sentiment in the text affects product demandproduct demand??

• How sentiment in the text affects How sentiment in the text affects informativenessinformativeness of reviews? of reviews?

Searching for reviews is different from general Web search.

– Reading only the review ranked at the top can be erroneous because it is only opinion of one person.

Improve the design of electronic markets

• Propose two ranking mechanisms for ranking product reviews:– a consumer-oriented ranking mechanism ranks the reviews according

to their expected helpfulness.

– a manufacturer-oriented ranking mechanism ranks the reviews

according to their expected effect on sales.

Data

DataData

Elements of a review and reviewer profile

Rating (number)->Moderate or Extreme Review

Number of helpful votes and total votes

Review posting date

Textual content-> ‘Subjective’ or ‘Objective’ Review and ‘Readability’

Reviewer information disclosure -> Real Name, Location, etc.

Overview

• Mix of products: Audio-Video, Digital Cameras.• Sample: March 2005 - April 2006

Summary of Amazon Data

• New: List Price, Amazon price, Sales rank, Release Date.• Valence, Volume of reviews

Measuring SubjectivityMeasuring Subjectivity

• Pang and Lee (2004)-technique that identifies which sentences in a text convey objective or subjective elements.• Sentences in product description: “Objective”• Sentences in reviews: “Subjective”

• A training set with two classes of documents:– A set of objective documents that contains the product descriptions of each

of the 1,000 products in our data set.– A set of subjective documents that contains randomly retrieved reviews.

• Train a classifier (n-gram based) using Dynamic Language Model classifier from LingPipe toolkit to distinguish between subjective and objective sentences in each review.– ‘Average Probability of Subjectivity’ for a review– ‘Standard deviation of Subjectivity’ for each review

• In our context: “Subjectivity” measures deviation of review from manufacturer-provided product information

Measuring ReadabilityMeasuring Readability

“Readability” measures amount of information in the review, and difficulty of reading the provided information

Current work:1. Measured number of sentences2. Measured length of review in words and characters3. Ratio of length in characters (or words) to number of

sentences.

Ongoing work: • Use ‘Readability’ statistics of the reviews

– ColemanLiau– Flesh Kinkaid Level– Gunning– SMOG Index

Impact of Sentiments and Sales

it it it it

it it i it

log(SALESRANK ) Log( AMAZONPRICE) log(ELAPSEDDATE ) AVGPROB

DEVPROB X

1 2 3 1

4 1 1

• Product level fixed effects

• We also ran first differences model using variations in the unit of time analysis (weekly, biweekly and monthly)

• Qualitative nature of results are robust to these specifications

• Derived the stylistic characteristics of each review. • Proceed to examine the economic impact of the

subjectivity (or objectivity) of the review.

Estimates for effect on sales rankEstimates for effect on sales rank

Independent Variable

Audio-Video Digital Camera

Average Subjectivity

-1.47***

(.72)1.27

(1.24)

Deviation of Subjectivity

-0.69(1.06)

-2.91***

(1.7)

Amazon Price 1.59***

(0.3)6.2*** (0.61)

Elapsed Date 0.12(0.07)

0.28**

(0.13)

Average Rating -0.016(0.02)

-0.01(0.03)

Log (Num. Reviews)

0.6*** (0.15)

1.08***

(0.2)

Readability 0.06(0.057)

-0.15*

(0.08)

R2 0.18 0.37

Effect of Sentiments on InformativenessEffect of Sentiments on Informativeness

Validation with Content AnalysisValidation with Content Analysis

• Asked two coders to classify 1,000 reviews (test set):– Is the review informative or not?– As influencing their purchasing decisions or not?

• Good inter-rater agreement (Kappa statistic of 0.73)

• For second question, using polychoric correlation agreement was strong (p < 0.05)

• Estimated regression coefficients using rest of reviews (training set)

• Predicted helpfulness and influence of the test reviews, using only text

F-measureF-measure

F-measure combines precision and recall in a single metric

Findings and ConclusionFindings and Conclusion

• Show that textual sentiment influences sales over and above the numeric and self-descriptive information that consumer reviews and reviewers display.

– Increasingly subjective reviews can increase product sales.

– A mixture of objective and subjective opinions with extreme subjective content can increase sales.

– A mixture of objective and subjective opinions with extreme subjective content can increases informativeness of reviews

– Increased `Readability’ can increases product sales and informativeness of reviews

• Our method can quickly identify reviews that are expected to be helpful to the users, and display them first.

– Improving the usefulness of the reviewing mechanism to the consumer in an electronic market.

– Manufacturers can understand which reviews are most impactful and use those product features for marketing promotions

Ongoing Work

• Combine the sentiments (positive, mixed or negative) with subjectivity analysis.– Negative reviews may increase sales if the reviews are

informative

• Incorporate the impact of all the past reviews of a reviewer.

• Examine more text-based variables, in detail (e.g. readability metrics).