Leveraging Joint Interactions for Credibility Analysis in...
Transcript of Leveraging Joint Interactions for Credibility Analysis in...
![Page 1: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/1.jpg)
Leveraging Joint Interactions for Credibility Analysis in News Communities
Subhabrata Mukherjee and Gerhard WeikumMax Planck Institute for Informatics
CIKM 2015
![Page 2: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/2.jpg)
Motivation
➢ Media plays a crucial role in public dissemination of information
➢ However, people believe there is substantial media bias in news in view of inter-dependencies and cross-ownerships of media companies and other industries (like energy)
➢ 4 out of 5 Americans among younger generations do not trust major news networks [Gallup poll, 2013]
➢ This work: Credibility Analysis of News Communities
![Page 3: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/3.jpg)
News Community
➢ A news community is a news aggregator site (e.g., reddit.com, digg.com, newstrust.net) where:➢ Users can give explicit feedback (e.g., rate, review, share) on the
quality of news➢ Interact (e.g., comment, vote) with each other
➢ However, this adds user subjectivity as users incorporate their own bias and perspectives in the framework
➢ Controversial topics create polarization among users which influence their evaluation
![Page 4: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/4.jpg)
Contributions
● A model to capture joint interaction between language, topics, users and sources leading to better prediction than the ones in isolation
● User expertise, source trustworthiness, language objectivity, topical perspective and article credibility mutually reinforce each other
● A supervised Conditional Random Field model that can capture these interactions, and handle real-valued ratings
![Page 5: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/5.jpg)
s1 s
1
d1
r11
r12
u1
u2
d2
r22
u2
y1
y2
C1 C
2
Source
Article
Review
User
ExampleFACTORS
![Page 6: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/6.jpg)
s1 s
1
d1
r11
r12
u1
u2
d2
r22
u2
y1
y2
C1 C
2
Source
Article
Review
User
Alternet.org(progressive/liberal)
Why do conservaties hate your children?
Ratings
Discussions(liberal vs.conservative)
Example
Topic: Climate
FACTORSInstantiation
![Page 7: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/7.jpg)
s1 s
1
d1
r11
r12
u1
u2
d2
r22
u2
y1
y2
C1 C
2
Viewpoint, Expertise
Why do conservaties hate your children?
Ratings
Discussions(liberal vs.conservative)
Example
Topic: Climate
Source
Article
Review
User
FACTORSFEATURES
![Page 8: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/8.jpg)
s1 s
1
d1
r11
r12
u1
u2
d2
r22
u2
y1
y2
C1 C
2
Viewpoint, Expertise
Emotionality, Discourse
Ratings
Discussions(liberal vs.conservative)
Example
Topic: Climate
Source
Article
Review
User
FACTORSFEATURES
![Page 9: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/9.jpg)
s1 s
1
d1
r11
r12
u1
u2
d2
r22
u2
y1
y2
C1 C
2
Viewpoint, Expertise
Emotionality, Discourse
Discussions(liberal vs.conservative)
Example
Ratings
Topic
Source
Article
Review
User
FACTORSFEATURES
![Page 10: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/10.jpg)
s1 s
1
d1
r11
r12
u1
u2
d2
r22
u2
y1
y2
C1 C
2
Viewpoint, Expertise
Emotionality, Discourse
Bias, Viewpoint, Expertise
Example
Topic
Ratings
Source
Article
Review
User
FACTORSFEATURES
![Page 11: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/11.jpg)
s1 s
1
d1
r11
r12
u1
u2
d2
r22
u2
y1
y2
C1 C
2
Source
Article
Review
User
Article Credibility Rating?
Trustworthiness
Objectivity
Credibility
Expertise
TaskFACTORSATTRIBUTES
![Page 12: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/12.jpg)
Credibility Analysis
➢ Given a set of news sources generating news articles, and users reviewing them on different qualitative aspects with mutual interactions:➢ Jointly rank the sources, articles, and users based
on their trustworthiness, credibility,and expertise
![Page 13: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/13.jpg)
Credibility of Statements in Health Communities[S. Mukherjee et al.: KDD‘14]
![Page 14: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/14.jpg)
s1 s
1
d1
r11
r12
u1
u2
d2
r22
u2
y1
y2
C1 C
2
Source
Article
Review
User
Objectivity
Language Features
Assertives, Factives, Hedges, Implicatives, Report, Discourse, Subjectivity etc.
1. M. Recasens, C. Danescu-Niculescu-Mizil, and D. Jurafsky. Linguistic models for analyzing and detecting biased language. In ACL, 2013.2. S. Mukherjee, G. Weikum, and C. Danescu-Niculescu-Mizil. People on drugs: Credibility of user statements in health communities. KDD, 2014.
![Page 15: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/15.jpg)
➢ Only 33% of the articles have explicit tags
➢ Use Latent Dirichlet Allocation to learn the latent topic distribution in the corpus of news articles
Topic Features
![Page 16: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/16.jpg)
Source Features
![Page 17: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/17.jpg)
Category Elements
Engagement answers, ratings (given / received), comments etc.
Agreement Inter-user agreement
Topics perspective and expertise
Interactions user-user, user-item, user-source
User Features
![Page 18: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/18.jpg)
s1 s
1
d1
r11
r12
u1
u2
d2
r22
u2
y1
y2
C1 C
2
Source
Article
Review
User
Article Credibility Rating?
Source Models
Article Language Model, Topic Model
Review Language Model, Topic Model
User Models
How to aggregate?
Given a factor, with its features, use Support Vector Regression to learn a model that will predict its rating for an article.
![Page 19: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/19.jpg)
Probability Mass Function for discrete labels:
Probability Density Function for continuous ratings:
Conditional Random Field
![Page 20: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/20.jpg)
Clique potential
Energy Function
User Potential
Source Potential Language Potential Topic Potential
Clique: source, article, <users>, <reviews>
![Page 21: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/21.jpg)
error of predictor SVR
partitions the user space
user expertise
![Page 22: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/22.jpg)
source trustworthiness
Energy Function
language objectivity topical perspective
![Page 23: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/23.jpg)
Σ needs to be positive definite for inverse to exist → {α, β, γ} > 0
Makes sense: predictor reliability should be positive
The joint p.d.f is a multivariate gaussian distribution
![Page 24: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/24.jpg)
Maximize log-likelihood with respect to log λk
instead of λk
Prediction is the expected value of the function given by the mean of the Multivariate Gaussian distribution:
Constrained optimization problem.Gradient ascent cannot be directly used.
![Page 25: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/25.jpg)
Experiments: NewsTrust
Data available at: http://www.mpi-inf.mpg.de/impact/credibilityanalysis/
![Page 26: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/26.jpg)
Predicting User Ratings
Users, Articles, Ratings
+Time
+Review Text
+Review Text andInteractions
1. Y. Koren. Factorization meets the neighborhood: A multifaceted collaborative filtering model. KDD, 2008.2. J. McAuley and J. Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. RecSys, 2013.3. J. J. McAuley and J. Leskovec. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In WWW, 2013.
![Page 27: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/27.jpg)
Predicting Article Credibility Ratings
![Page 28: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/28.jpg)
Predicting Article Credibility Ratings
![Page 29: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/29.jpg)
Predicting Article Credibility Ratings
![Page 30: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/30.jpg)
Predicting Article Credibility Ratings
![Page 31: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/31.jpg)
Ranking Trustworthy News Sources
Ranking Expert Users:
![Page 32: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/32.jpg)
Sample Output: Most and Least Trust Sources on Sample Topics
![Page 33: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/33.jpg)
Conclusions
➢ Joint interaction between language, topics, users and sources lead to better prediction in multiple tasks
➢ User expertise, source trustworthiness, language objectivity, topical perspective and article credibility mutually reinforce each other
![Page 34: Leveraging Joint Interactions for Credibility Analysis in ...resources.mpi-inf.mpg.de/impact/credibility... · A model to capture joint interaction between language, topics, users](https://reader034.fdocuments.us/reader034/viewer/2022050601/5fa85f15aea7a5747e208fad/html5/thumbnails/34.jpg)
Ongoing Work
➢ Analyze temporal evolution of these factors
➢ Communities are inherently dynamic in nature➢ Source trustworthiness, and user expertise change
with time
➢ To this end we propose an Experience-aware Item Recommendation for Evolving Review Communities, ICDM 2015.