Carolyn Penstein RoséLanguage Technologies Institute
and Human-Computer Interaction Institute
With funding from the National Science Foundation and the Office of Naval Research
Acknowledgements
DongNguyen
Elijah Mayfield
HuaAi
Rohit Kumar
IrisHowley
Nguyen, D., Mayfield, E., & Rosé, C. P. (2010). An analysis of perspectives in interactive settings, in Proceedings of the KDD Workshop on Social Media Analytics.
Kumar, R. & Rosé, C. P. (2010). Engaging learning groups using Social Interaction Strategies, In Proceedings of the North American Chapter of the Association for Computational Linguistics.
Howley, I., Mayfield, E. & Rosé, C. P. (to appear). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.
Ai, H., Kumar, R., Nguyen, D., Nagasunder, A., Rosé, C. P. (2010). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, in Proceedings of Intelligent Tutoring Systems.
DongNguyen
Elijah Mayfield
HuaAi
Rohit Kumar
IrisHowley
For more information about my group:http://www.cs.cmu.edu/~cprose
OutlineMotivation from Opinion MiningTheoretical framework from Rhetoric and Discourse
AnalysisStudy one: Political bias in a political discussion forumStudy two: Goal orientation in chat based design
discussionsCurrent Directions
OutlineMotivation from Opinion MiningTheoretical framework from Rhetoric and Discourse
AnalysisStudy one: Political bias in a political discussion forumStudy two: Goal orientation in chat based design
discussionsCurrent Directions
Typical paradigm for sentiment analysis of product reviews:Make a prediction based on text of single reviews taken out
of context
Some evidence of group effects in product review blogs based on numerical ratings (Wu et al., 2008)
KEY ASSUMPTION: language is a reflection of the speaker’s perspective
Are product reviews conversational?
KEY ASSUMPTION: language is a reflection of the speaker’s perspective
Work towards weakening the assumptionSometimes use syntactic cues to reverse polarity on some terms
(Somasunderan & Wiebe, 2009; Wijaya & Bressan, 2008)Factoring out the effect of context rather than modeling it
Aggregation over all reviews posted by the same individual Taking opinions of similar individuals into account (e.g.,
collaborative filtering)
Falls short of modeling conversational aspects of product reviews
KEY ASSUMPTION: language is a reflection of the speaker’s perspective
Are product reviews conversational?
KEY ASSUMPTION: language is a reflection of the speaker’s perspective
Are product reviews conversational?“After many MANY weeks of research, gathering
information from several sites, reviews etc I decided that the Britax Boulevard was definitely the safest bet available on the market. The things that sold me: All the safety gadgets that other seats don't have like the side impact wings, the HUGS system, the LATCH system and 5 point harness and also the fact that it lasts up to 29Kg. “
Are product reviews conversational?“I did most of my research on the net, picking my
top 3 choices I went and had a look at them in the shops. I looked at one the Graco Comfort Sport, the Britax Boulevard and the Decathlon and Marathon seats. By far it seems that Britax have the upper hand safely wise on the market, many professional reviews and crash tests agree on this so Britax was the clear choice for us. “
Are product reviews conversational?“I have the seat front facing in my Camry (2007) I
worried about the size of the chair from reading other reviews but that is NO problem in my car, my son has plenty of leg room and can see perfectly out the window.”
Are product reviews conversational?
http://www9.georgetown.edu/faculty/irvinem/theory/Bakhtin-MainTheory.html
OutlineMotivation from Opinion MiningTheoretical framework from Rhetoric and
Discourse AnalysisStudy one: Political bias in a political discussion forumStudy two: Goal orientation in chat based design
discussionsCurrent Directions
Discourse and IdentityIdentity is reflected in the way we present ourselves
in conversational interactionsReflects who we are, how we think, and where we
belongAlso reflects how we think of our audience
ExamplesRegional dialect: shows my identification with where
I am from, but also shows I am comfortable letting you identify me that way
Jargon and technical terms: shows my identification with a work community, but also shows I expect you to be able to relate to that part of my life
Level of formality: shows where we stand in relation to one another
Explicitness in reference: shows whether I am treating you like an insider or an outsider
Discourse and Identity
Lave & Wenger, 1991
Discourse is text above the clause level (Martin & Rose, 2007)
A Discourse is an ongoing conversation [type]Socialization is the process of joining a
Discourse (Lave & Wenger, 1991; Sfard, 2010)
We join Discourses that match our core identity (de Fina, Schiffrin, & Bamberg, 2006)
In moving from the periphery to the core of a Discourse community, we sound more and more like the community (Arguello et al., 2006)
A discourse is one instance of it [token]All discourses contain echoes of
previous discourses (Bakhtin, 1983)
Lakoff & Johnson, 1980
Metaphors Structure our Experience
We describe arguments using terms related to warUsing a typical war ‘script’ to
structure a story about an argument
We orient towards arguments as though they were warsOur conversational partner is
our opponentWe may feel that we won or
lostWe may feel wounded as a
result
Discourses, Frames, and Metaphors
Frame: A portion of a discourse belonging to distinct Discourse
Metaphor : One linguistic device that can be used to define a set of discourse practices that constitute a frame
Topic models: a technical approach that makes sense for identifying frames within a discourse
A discourse could be drawn from a mixture of DiscoursesWithin the same conversation, we may wear a
variety of “hats”E.g., the same discourse with a co-worker may
contain exchanges pertaining to our relationship as colleagues and others to our relationship as friends
Model of Communication from Rhetoric
Implied author: Communication style is a projection of identityImpression management, not
necessarily the ground truthImplied reader: What we assume
about who is listeningReal assumptions, possibly incorrectWhat we want recipients or
overhearers to think are our assumptions
Reader: may or may not understand the text the way it was intended
Author
ImpliedAuthor
ImpliedReader
Text
Effect
Reader
Engagement: Social positioning in conversational style
The message: Most contributions express some content
Implied author: How I phrase it says something about my stance with respect to that content
Implied reader: Also says something about what I assume is your stance and my stance in relation to you
Reader: The hearer may respond either to the message or its positioning
Author
ImpliedAuthor
ImpliedReader
Text
Effect
Reader
Engagement: Social positioning in conversational style
Speaker 1: I want chocolate for dessert.Speaker 2: [you can’t have chocolate]
Options with different implications about author and reader You can’t have chocolate. You’re allergic to chocolate. You’re allergic to chocolate, so eating it
would be a bad idea. Your mom said you’re allergic to chocolate Having chocolate might be a poor choice for
you. Having chocolate might be a poor choice for
you for a great number of reasons.
Author
ImpliedAuthor
ImpliedReader
Text
Effect
Reader
Even Scientific Writing is Social and Conversational
Implied author: Rhetorical style in academic writing gives an impression of who we are as researchers
Implied reader: targeting writing to community standardsAbstracts and literature reviews
position us in research communityReader: research papers teach us
both about the content of our field and its politics
Author
ImpliedAuthor
ImpliedReader
Text
Effect
Reader
OutlineMotivation from Opinion MiningTheoretical framework from Rhetoric and Discourse
AnalysisStudy one: Political bias in a political discussion
forumStudy two: Goal orientation in chat based design
discussionsCurrent Directions
Bias Estimation
Start with LDA model of politics dataset with 15 topicsThen separate the texts into two collections, one left
affiliated, and one right affiliatedWe then have a Left model and a Right model
We can then compute a rank for each word w in each topic t in each modelIntuition: a word is more distinguishing for a particular point
of view if it has a high probability within the associated model and a low probability in the opposite model
Bias(w,t) = log(rankright(w,t) + 1) – log(rankleft(w,t) + 1)The bias of a text is the average bias over the terms within
the text Left scores positive, right scores negative
Qualitative Analysis
Terror Language (Right): evokes emotional response to thread of attack. Define target as evil and as a threat. Provokes a defensive posture.
Imperialist rhetoric (Right): racial prejudice, attitude of superiority.
Web of concern (Left): focus on opposition as individuals with a culture and history, concern for wellbeing of all people, focus on potential negative effects of war
Quantitative AnalysisRight BiasLeft Bias
Score of posterScore of quoted
messageScore of full postScore of words
that appear in both messages
Score of words that appear only in quoted message
Score of words that appear only in the post
Investigation of Quoting behavior
Negative correlation between words only in quoted message and words only in post (r=-0.1, p < 0.05)
Positive correlation between score quoted words and score of the whole post (r=0.18, p < 0.02)
Score of words only in post are significantly more reflective of the affiliation of the poster than that of the author of the quoted messageSimilar result with score of words only in quote with
affiliation of author of quoted message
Investigation of Quoting behavior
Investigation of Quoting behavior
Which words are quoted?
by pointing out the inflation of Saddam’s body count by neocons in an effort to further vilify him and thus further justify our invasion we are not DEFENDING saddam....just pointing out how neocons rarely
let facts get in the way of a good war.
by pointing out the inflation of Saddam’s body count by neocons in an effort to further vilify him and thus further justify our invasion we are not DEFENDING saddam....just pointing out how neocons rarely
let facts get in the way of a good war.
So wait, how many do you think Saddam killed or oppressed? You’re trying to make him look better than he actually was.
You’re the one inflating the casualties we’ve caused! Seriously, what estimates (with a link) are there that we’ve killed over 100,000 civilians. Not some crack pot geocities page either.
So wait, how many do you think Saddam killed or oppressed? You’re trying to make him look better than he actually was.
You’re the one inflating the casualties we’ve caused! Seriously, what estimates (with a link) are there that we’ve killed over 100,000 civilians. Not some crack pot geocities page either.
Thread level analysis
Effect of initial postCorrelation between score thread (without first post)
and first post = 0.210 (p<0.01)Effect of Prior posts
Aggregate score of previous postsDifference in score of current post and average score of
user Small correlation (r=0.133, p < 0.01) indicating that
users talk more left than they usually do when previous posts are left and similarly for right.
Overview of Findings
Quotes from opposite point of view include the words that are less strongly associated with the opposite perspective
Because of quotes, displayed bias shifts towards the bias of the person to whom the message is directed
Personal bias of the speaker is most strongly represented by non-quoted portions of text
The effect of a post extends past just the immediate response
OutlineMotivation from Opinion MiningTheoretical framework from Rhetoric and Discourse
AnalysisStudy one: Political bias in a political discussion forumStudy two: Goal orientation in chat based design
discussionsCurrent Directions
Collaborative Design TaskGoal: Design a power plantCompeting Student Goals:
Power: Design a power plant that achieves maximum power output
Green: Design a power plant that has the minimum impact on the environment
Competing goals encourages deeper discussionExploration of design spaceExplicit articulation of reasoning
30> Experimental Design >Task
31
Classroom Study
Chat room style interaction ConcertChat
106 studentsCMU undergrads in ME
Classroom sessionDuring the semester• Instruction Pretest
Session Posttest Questionnaire
31> Experimental Design >Study Procedure
Experimental Design
HypothesesStudents will be more engaged with agents displaying
social behaviorsStudents will be sensitive to tutor goal orientationInteraction effect
FrequentGreen
InfrequentGreen
FrequentNeutral
NonePower
NoneNeutral
FrequentPower
InfrequentNeutral
InfrequentPower
NoneGreen• Social Behavior
– Frequent, Infrequent, None
• Goal Alignment– Green, Power, Neutral
32> Experimental Design >Manipulations
An Example of Displaying Bias
33> Experimental Design >Agent Design
Green BiasGreen: What is bad about increasing heat input to the cycle is that
more waste heat is rejected to the environment.Neutral and Power: Increasing heat input to the cycle increases
waste heat rejected to the environment.
Power Bias:Power: What is good about increasing heat input to the cycle is
that more power output is produced.Neutral and Green: Increasing heat input to the cycle increases
power output produced.
34
Example of Social Behaviors
34> Experimental Design >Agent Design
1. Showing Solidarity: Raises other's status, gives help, reward
2. Showing Tension Release: Jokes, laughs, shows satisfaction
3. Agreeing: Shows passive acceptance, understands, concurs, complies
Tutor: Let’s Introduce ourselves. My name is Avis. Tutor: Be nice to your teammates!
Tutor: I’m happy to work with our team :-)
Tutor: m-hmm (showing attention)
Adapted from Bales’ IPA (Bales, 195)
Experimental Design
HypothesesStudents will be more engaged with agents displaying
social behaviorsStudents will be sensitive to tutor goal orientationInteraction effect
FrequentGreen
InfrequentGreen
FrequentNeutral
NonePower
NoneNeutral
FrequentPower
InfrequentNeutral
InfrequentPower
NoneGreen• Social Behavior
– Frequent, Infrequent, None
• Goal Alignment– Green, Power, Neutral
35> Experimental Design >Manipulations
36
Measuring Student BiasUsing a topic modeling tool – ccLDA [Paul and Girju, 2009]
36> Experimental Design >Displaying Bias
CorpusCollection1 Collection2
Topic1
Topic 2
Topic 3
Example of Extracted Topics Heat quality right max decrease possible goes efficiency need gas graph say natural want goal fuel Tmax min sounds temp going friendly turbine kpa mean
11000 values different makes larger graphs bit green large kind produce hate steam team step solid 6574 split bored nat geo instead happens plant love
yah blades sir dunno kk x85 rejected guessing starts FINAL life helping compromise nd depends corresponding teammate stays tmin new hard sitting afk tmax500 bec
power decreases nuclear make 85 cycle work guess high pmin want pmax wait 570 lower green 40 tmax Pmin value low best point pick environment
low 500 12800 sort 1 tutors effeciency 440 coool ecofriendly half fun 105 Nuclear sweeet maximized cooler question boy 6000 worked creepy Goes 16250 maxes
generates makes 085 different 7000 12000 qdot becuase decreasing click leads liquid gues doubt 10790 meet POWER 6574 DESIGN transfer hope Qin 11000 discussion km
TOPIC 1 TOPIC 2Background Green Power Background Green Power
37> Experimental Design >Displaying Bias
Bias Measurement MetricsMax Topic-word bias: count the number of words in the list of
the N most strongly associated words, and take the maximum across topics
Average Topic-word bias: count the number of words in the list of the N most strongly associated words, and take the average across topics
Weighted Topic-Word Average bias: Same but weight each word by its association within the background model first
All three measures highly correlated both for Green and for Power perspectives
Students in the Green condition got higher Green scores on average than Power scores and vice versa in the Power conditionOnly statistically significant for the first two metrics
Measuring Influence
Within pairs, the Green score of the Green student and the Green score of the Power student were significantly correlatedSame story for Power scoresResult consistent with analysis of Politics dataset
39> Experimental Design >Displaying Bias
Operationalization of Authoritativeness
Negotiation coding scheme (Martin & Rose, 2007, Chapter 7)
Agreement on K1/K2/Other .72 Kappa
Coded all transcripts from Infrequent Social conditionAuthoritativeness score
= K1/[K1 + K2]Within pairs, one
Authoritative student and one NonAuthoritative student
Balance Effect• Alignment
• Align: Authoritative partner shares affiliation with agent
• Neutral: Agent is neutral• NoAlign: Non-Authoritative partner shares
affiliation with agent• Affiliated agent has polarizing effect on
displayed bias• Difference in bias scores was significantly higher
in conditions with affiliated agents• Direction of polarization depends on alignment
• Balance Effect• Authoritative student shows less of his own
bias when he’s in the minority• NonAuthoritative student is less non-
authoritative when he’s in the minority
Overview of Findings
Topic models can display differences in goal orientation in chat data
Confirmation of influence of partner speech on displayed bias
Complex relationship between personal orientation, authoritativeness, and ingroup/outgroup effects
OutlineMotivation from Opinion MiningTheoretical framework from Rhetoric and Discourse
AnalysisStudy one: Political bias in a political discussion forumStudy two: Goal orientation in chat based design
discussionsCurrent Directions
Current DirectionsFull circle for opinion miningContinuing to operationalize multiple dimensions of
relational codesHowley, I., Mayfield, E., & Rosé (to appear). Linguistic Analysis
Methods for Studying Small Groups, in Hmelo-Silver, O’Donnel, & Chan (Eds.) International Handbook of Collaborative Learning, Taylor & Francis, Inc.
Collaboration with Bob Kraut: investigating how exchange of social support is reflected in relational codes
Collaboration with Bhiksha Raj: investigating evidence of relational codes like Negotiation in speech
CHI submission in progress: Analysis of effects of bullying behavior on distribution of relational codes and learning
New grant!: studying the emergence of leadership in ad-hoc teams
Computational Models of Discourse AnalysisFocus on literature from the field of Discourse
AnalysisInvestigating issues such as Conversational Structure,
Attitude, Perspective, Persuasion and Positioning Critical reflection on the state-of-the art in language
technologiesHands-on programming assignments, fun contests
Carolyn Penstein Roséhttp://www.cs.cmu.edu/~cprose
[email protected] Center 5415
Top Related