Text Summarization of Review...
Transcript of Text Summarization of Review...
![Page 1: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/1.jpg)
Text Summarization of Review Sentiments
Eric JensenSummize, Inc.
![Page 2: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/2.jpg)
![Page 3: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/3.jpg)
Outline
!Opinions on the web!Opinion mining!Text summarization
"The problem"Proposed algorithm"Results
!Conclusions
![Page 4: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/4.jpg)
Growth of Amazon, IMDB, and Blogs
0
500K
1.0M
1.5M
2.0M
2.5M
3.0M
3.5M
1999 2001 2003 2005 2007
UserReviews
BlogReviews
![Page 5: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/5.jpg)
Opinions on the web
Length
Focus
Amazon Users
Blogs
Consumer Reports
Four Word Film Review
Yahoo Answers
![Page 6: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/6.jpg)
Support (or lack of?)
40%
50%
60%
70%
80%
90%
100%
1 11 21 31 41 51 61 71 81 91 101 111 121Number of Review s
Cum
ulative Proportion
![Page 7: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/7.jpg)
How many are you willing to read?
![Page 8: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/8.jpg)
Opinion mining
!Sentiment analysis!Facet mining!Text summarization
![Page 9: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/9.jpg)
Sentiment analysis! (Pang EMNLP 2002, Dave, et. al WWW 2003)
I Am Legend
�I won't review the movie because this has already been done. What I will rate is the 2-disc �Special Edition� of this movie�Overall, I feel this 2-disc edition is not worth the extra money it costs.�
![Page 10: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/10.jpg)
Facet mining (Hu and Liu KDD 2004, Popescu and Etzioni EMNLP 2005, Titov and McDonald WWW 2008)
!Digital camera"Resolution"Zoom"User interface
! I Am Legend"Acting"Special effects"2-disc special edition?
![Page 11: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/11.jpg)
Text summarization
The problem: understand the prevailing sentiments as quickly as possible
!Leverage the ratings users provide to produce more meaningful summaries
!Don�t restrict to fixed categories/facets!Why did the users rate it this way
![Page 12: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/12.jpg)
Example
I Am Legend
riveting movie �hollywood ending �amazing story � excellent character � riveting performance� dark sci-fi � grotesque film
![Page 13: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/13.jpg)
Experimentation
!Dataset!Evaluation!Baseline!Results!Consensus Building
![Page 14: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/14.jpg)
Experimentation: Dataset
!Amazon and IMDB!10 million user reviews!3.6 million products!Books, movies, music, and others
![Page 15: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/15.jpg)
Evaluation
!Sampled 30 products"Stratified by category"Minimum of 10 reviews each
!Task: ideal 10-word summary of the prevailing sentiments about that product"Mix positive and negative in appropriate ratio"Arbitrary length phrases
!E.g. vacuum cleaner: high suction, heavy, do not buy
![Page 16: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/16.jpg)
Evaluation: Metrics
! Text Analysis Conference (formerly DUC)
! Overlap of reference summaries highly correlated with manual evaluation (Lin & Hovy HLT-NAACL 2003)
( )
( )n
n
match ngram reference
ngram reference
Count gramROUGE N
Count gram∈
∈
− =∑
∑
![Page 17: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/17.jpg)
Framework
riveting movie �hollywood ending �amazing story � excellent character � riveting performance� dark sci-fi � grotesque film
Input Output
![Page 18: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/18.jpg)
Baseline: Adapted facet-oriented mining (Hu and Liu KDD 2004)
1. Identify noun phrases and treat adjacent adjectives as opinion words
2. Rank noun phrases by TFxIDF
3. Choose top opinion word by frequency
4. Choose top summary phrases by frequency
- 3 & 4 our adaptation
![Page 19: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/19.jpg)
Proposed algorithm
1. Identify each opinion word and treat the following word as a �facet� word
2. Rank facet words by frequency
3. Choose top opinion word by frequency
4. Choose top phrases by frequency
![Page 20: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/20.jpg)
Results
+55.03%
0.0910.0880.107Summize ROUGE-SU4
0.0590.0540.161Facets ROUGE-SU4
+36.25%
0.0450.0440.050Summize ROUGE-2
0.0330.0250.105Facets ROUGE-2
+26.81%
0.2730.2630.293Summize ROUGE-10.2150.1890.329Facets ROUGE-1
F0.5RecallPrecisionMethod / Metric
![Page 21: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/21.jpg)
Consensus Building
100
101
102
103
104
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
revi ew cnt
pro
babi
lity
fract ion of productscluster probabilit y
![Page 22: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/22.jpg)
Conclusions
!Number of opinions on the web are growing faster than anyone wants to read
!Text summarization reveals the why behind the ratings
!Facets do not capture the ideal summaries (sentiment-oriented ones are 26% closer)
!Scaling is both a problem and an opportunity
![Page 23: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/23.jpg)
Future Directions
!Scale to more and more reviews!Analyze opinions from unstructured
sources (blogs, twitters, etc.)
![Page 24: Text Summarization of Review Sentimentslintool.github.io/UMD-courses/bigdata-2008-Spring/speakers/2008-0… · (Pang EMNLP 2002, Dave, et. al WWW 2003) I Am Legend fiI won’t review](https://reader031.fdocuments.us/reader031/viewer/2022040920/5e9882d594d617090040690f/html5/thumbnails/24.jpg)
Plugging my other work
!Semi-automatic evaluation (ACM TOIS �07)
!Query classification (ACM TOIS �07)!Query log analysis (SIGIR �04)