Modeling Information Seeking Behavior in Social Media

Eugene AgichteinIntelligent Information Access Lab (IRLab)

Eugene Agichtein, Emory University, IR Lab 2

Intelligent Information Access Lab (IRLab)

Qi Guo (3rd year Phd)

Ablimit Aji (2nd year PhD)

• Modeling information seeking behavior• Web search and social media search• Text and data mining for medical informatics and

public health

In collaboration with: - Beth Buffalo (Neurology)- Charlie Clarke (Waterloo)- Ernie Garcia (Radiology)- Phil Wolff (Psychology)- Hongyuan Zha (GaTech)

1st year graduate students: Julia Kiseleva, Dmitry Lagun, Qiaoling Liu, Wang Yu

Yandong Liu (2nd year Phd)

Eugene Agichtein, Emory University, IR Lab 3

Online Behavior and Interactions

Information sharing: blogs, forums, discussions

Search logs: queries, clicks

Client-side behavior: Gaze tracking, mouse movement, scrolling

Research Overview

Eugene Agichtein, Emory University, IR Lab

44

Information sharing

Health Informatics

Cognitive Diagnostics

Intelligent search

Discover Models of Behavior(machine learning/data mining)

Eugene Agichtein, Emory University, IR Lab 5

Key Challenges for Web Search

• Query interpretation (infer intent)

• Ranking (high dimensionality)

• Evaluation (system improvement)

• Result presentation (information visualization)

Eugene Agichtein, Emory University, IR Lab 6

Contextualized Intent Inference

• SERP text• Mouse trajectory, hovering/dynamics• Scrolling• Clicks

Eugene Agichtein, Emory University, IR Lab 7

Research Intent

Eugene Agichtein, Emory University, IR Lab 8

Purchase Intent

Eugene Agichtein, Emory University, IR Lab 9

Relationship between behavior and intent?

• Search intent is contextualized within a search session

• Implication 1: model session-level state • Implication 2: improve detection based on client-

side interactions

Eugene Agichtein, Emory University, IR Lab 10

Model: Linear Chain CRF

Eugene Agichtein, Emory University, IR Lab 11

Results: Ad Click Prediction

• 200%+ precision improvement (within mission)

Research Overview

Eugene Agichtein, Emory University, IR Lab

1212

Information sharing

Health Informatics

Cognitive Diagnostics

Intelligent search

Discover Models of Behavior(machine learning/data mining)

Finding Information Online (Revisited)

13

Next generation of search: Algorithmically-mediated information exchange

CQA (collaborative question answering):• Realistic information exchange

• Searching archives

• Train NLP, IR, QA systems

• Study of social behavior, norms

Content quality, asker satisfaction

Current andfuture work

Goal: Hybrid Human-Powered Search

1414

Eugene Agichtein, Emory University, IR Lab 15

Talk Outline

Overview of the Emory IR Lab

Intent-centric Web Search

Classifying intent of a query

Contextualized search intent detection

16

(Text) Social Media Today

Published: 4Gb/day

Social Media: 10Gb/Day

Technorati+Blogpulse120M blogs2M posts/day

Twitter: since 11/07:2M users3M msgs/day

Facebook/Myspace: 200-300M usersAvg 19 m/day

Yahoo Answers: 90M users, 20M questions, 400M answers[Data from Andrew Tomkins, SSM2008 Keynote]

Yes, we could read your blog. Or, you could tell us about your day

18

19

Total time: 7-10 minutes, active “work”

Someone must know this…

21+1 minute

+7 hours: perfect answer

Update (2/15/2009)

23

24

http://answers.yahoo.com/question/index;_ylt=3?qid=20071008115118AAh1HdO

25

Finding Information Online (Revisited)

26

Next generation of search: Algorithmically-mediated information exchange

CQA (collaborative question answering):• Realistic information exchange

• Searching archives

• Train NLP, IR, QA systems

• Study of social behavior, norms

Content quality, asker satisfaction

Current andfuture work

(Some) Related Work

• Adamic et al., WWW 2007, WWW 2008:– Expertise sharing, network structure

• Elsas et al., SIGIR 2008: – Blog search

• Glance et al.: – Blog Pulse, popularity, information sharing

• Harper et al., CHI 2008, 2009: – Answer quality across multiple CQA sites

• Kraut et al.: – community participation

• Kumar et al., WWW 2004, KDD 2008, …: – Information diffusion in blogspace, network evolution

SIGIR 2009 Workshop on Searching Social Mediahttp://ir.mathcs.emory.edu/SSM2009/

27

Finding High Quality Content in SM

• Well-written• Interesting• Relevant (answer)• Factually correct• Popular?• Provocative?• Useful?

28

As judged by professional editors

E. Agichtein, C. Castillo, D. Donato, A. Gionis, and G. Mishne, Finding High Quality Content in Social Media, in WSDM 2008

Social Media Content Quality

29

E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne, Finding High Quality Content in Social Media, WSDM 2008

quality

3030

31

How do Question and Answer Quality relate?

3232

3333

3434

3535

Community

36

Link Analysis for Authority Estimation

37

Question 1

Question 2

Answer 5

Answer 1

Answer 2

Answer 4

Answer 3

User 1

User 2

User 3

User 6

User 4

User 5

Answer 6

Question 3

User 1

User 2

User 3

User 6

User 4

User 5

Kj

jAiH..0

)()(

Mi

iHjA..0

)()(

Hub (asker) Authority (answerer)

Qualitative Observations

HITS effective

HITS ineffective

38

3939

Random forest classifier

Result 1: Identifying High Quality Questions

40

Top Features for Question Classification

• Asker popularity (“stars”)

• Punctuation density

• Question category

• Page views

• KL Divergence from reference LM41

Identifying High Quality Answers

42

Top Features for Answer Classification

• Answer length

• Community ratings

• Answerer reputation

• Word overlap

• Kincaid readability score43

Finding Information Online (Revisited)

44

• Next generation of search: • human-machine-human

• CQA: a case study in complex IRContent quality• Asker satisfaction• Understanding the interactions

Dimensions of “Quality”

• Well-written• Interesting• Relevant (answer)• Factually correct• Popular?• Timely?• Provocative?• Useful?

45

As judged by the asker (or community)

Are Editor Labels “Meaningful” for CGC?

• Information seeking process: want to find useful information about topic with incomplete knowledge– N. Belkin: “Anomalous states of knowledge”

• Want to model directly if user found satisfactory information

• Specific (amenable) case: CQA

Yahoo! Answers: The Good News

• Active community of millions of users in many countries and languages

• Effective for subjective information needs– Great forum for socialization/chat

• Can be invaluable for hard-to-find information not available on the web

47

48

Yahoo! Answers: The Bad News

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10

49

May have to wait a long time to get a satisfactory answer

May never obtain a satisfying answer

1. FIFA World Cup2. Optical3. Poetry4. Football (American)5. Soccer6. Medicine7. Winter Sports8. Special Education9. General Health Care10. Outdoor RecreationTime to close a question (hours)

Predicting Asker Satisfaction

Given a question submitted by an asker in CQA, predict whether the user will be satisfied with the answers contributed by the community.

– “Satisfied” :• The asker has closed the question AND• Selected the best answer AND• Rated best answer >= 3 “stars” (# not important)

– Else, “Unsatisfied

50

Yandong Liu Jiang Bian

Y. Liu, J. Bian, and E. Agichtein, in SIGIR 2008

51

ASP: Asker Satisfaction Prediction

asker is satisfied

asker is not satisfied

TextCategory

Answerer History

Asker History

Answer

Question

Wikipedia

News

Classifier

52

Experimental Setup: Data

Questions

Answers Askers Categories

% Satisfied

216,170 1,963,615

158,515

100 50.7%

Crawled from Yahoo! Answers in early 2008

“Anonymized” dataset available at: http://ir.mathcs.emory.edu/shared/

1/2009: Yahoo! Webscope : “Comprehensive” Answers dataset: ~5M questions & answers.

Satisfaction by Topic

Topic Questions

Answers

A per Q

Satisfied

Asker rating

Time to close by asker

2006 FIFA World Cup

1194 35,659

329.86

55.4%

2.63 47 minutes

Mental Health

151 1159 7.68 70.9%

4.30 1.5 days

Mathematics

651 2329 3.58 44.5%

4.48 33 minutes

Diet & Fitness

450 2436 5.41 68.4%

4.30 1.5 days

53

54

Satisfaction Prediction: Human Judges

• Truth: asker’s rating• A random sample of 130 questions• Researchers

– Agreement: 0.82 F1: 0.45 2P*R/(P+R)

• Amazon Mechanical Turk– Five workers per question. – Agreement: 0.9 F1: 0.61 – Best when at least 4 out of 5 raters agree

Performance: ASP vs. Humans (F1, Satisfied)

Classifier With Text Without Text Selected Features

ASP_SVM 0.69 0.72 0.62

ASP_C4.5 0.75 0.76 0.77

ASP_RandomForest 0.70 0.74 0.68

ASP_Boosting 0.67 0.67 0.67

ASP_NB 0.61 0.65 0.58

Best Human Perf 0.61

Baseline (random)

0.66

55ASP is significantly more effective than humans

Human F1 is lower than the random baseline!

Top Features by Information Gain

• 0.14 Q: Askers’ previous rating• 0.14 Q: Average past rating by

asker• 0.10 UH: Member since (interval)• 0.05 UH: Average # answers for by

past Q• 0.05 UH: Previous Q resolved for the

asker• 0.04 CA: Average asker rating for

category• 0.04 UH: Total number of answers

received…

56

57

“Offline” vs. “Online” Prediction

• Offline prediction (AFTER answers arrive)– All features( question, answer, asker & category)– F1: 0.77

• Online prediction (BEFORE question posted)– NO answer features– Only asker history and question features (stars,

#comments, sum of votes…)– F1: 0.74

Personalized Prediction of Satisfaction

Same information != same usefulness for different searchers!

Personalization vs. “Groupization”?

58

Y. Liu and E. Agichtein, You've Got Answers: Personalized Models for Predicting Success in Community Question Answering, ACL 2008

Example Personalized Models

59

Outline

60

• Next generation of search: • Algorithmically mediated information exchange

• CQA: a case study in complex IRContent qualityAsker satisfaction

Current Work (in Progress)

• Partially supervised models of expertise(Bian et al., WWW 2009)

• Real-time CQA

• Sentiment, temporal sensitivity analysis

• Understanding Social Media dynamics

Answer Arrival

62

5 10 15 20 25 30 35 40 45 50 55 600

100000

200000

300000

400000

500000

600000

700000

573086

378227

146845

7226046364 34573 27322 23194 19952 17260 15481 13985

First Hour (69%)

Time in minutes

Answer number arrived in < T

Exponential Decay Model [Lerman 2007]

Factors Influencing Dynamics

Example: Answer Arrival | Category

Subjectivity

Answer, Rating Arrival

Preliminary Results: Modeling SM Dynamics for Real-Time Classification

• Adapt SM dynamics models to classificatione.g.: predict ratings

feature value:

Outline

69

• Next generation of search: • Algorithmically mediated information exchange

• CQA: a case study in complex IRContent qualityAsker satisfactionUnderstanding social media dynamics

Eugene Agichtein, Emory University, IR Lab 70

Question Urgency

Problem – a growing volume of questions competing for visibility

• Time-sensitive (urgent) questions pushed out by newer questions

• Delayed responses may become useless to seeker – wastes site resources and responders’ time

Goal: Query Processing over Web and Social Systems

7171

Takeaways

Robust machine learning over behavior data system improvements, insights into behavior

Contextualized models for NLP and text mining system improvements, insights into interactions

Mining social media: potential for transformative impact for IR, sociology, psychology, medical informatics, public health, …

72

References • Modeling web search behavior [SIGIR 2006, 2007]• Estimating content quality [WSDM 2008]• Estimating contributor authority [CIKM 2007]• Searching CQA archives [WWW 2008, WWW 2009]• Inferring asker intent [EMNLP 2008]• Predicting satisfaction [SIGIR 2008, ACL 2008, TKDE]• Coping with spam [AIRWeb 2008]

More information, datasets, papers, slides:http://www.mathcs.emory.edu/~eugene/

Eugene Agichtein, Emory University, IR Lab 74

Thank you!

• Yandex (for hosting my visit)

Supported by: