1. Supporting Social Deliberative Skills Online: the Effects of Reflective Scaffolding Tools

2. A Prototype Facilitators Dashboard: Assessing and visualizing dialogue quality in online deliberation for education and work

3. Text Analysis of Deliberative Skills in Undergraduate Online Dialogue: Using L1 Regularized Logistic Regression

with Psycholinguistic Features

Tom Murray, Xiaoxi Xu, Beverly Woolf, Leah Wing, Lynn Stephens,

Natasha Shrikant, Lori Clarke, Lee Osterweil

EEE 2013 & HICC 2013

Supporting Social Deliberative Skills in Online Contexts

2

Project Overview

Overview

1. Background–Social Deliberative Skills2. Classroom Studies3. Facilitator Dashboard4. Automated Text Analysis/Classification5. Conclusions

1. Background on Social Deliberative Skills

Group & CollaborativeLearning/Wo

rk

Conflict Resolution

Meaning Negotiation

Problem solving

Planning

Brainstorming

Inquiry

Decision making

Knowledge building

Group dynamics

(form, storm, norm)

Peer help/tutoring

Meaning Negotiation/Conflict Resolution

Scope: Support the skills needed to bridge different perspectives to build mutual understanding and mutual regard

Educational Priorities

• King & Baxter (2005) note that “In times of increased global interdependence, producing interculturally competent citizens who can engage in informed, ethical decision-making when confronted with problems that involve a diversity of perspectives is becoming an urgent educational priority [however, these skills] are what corporations find in shortest supply among entry-level candidates"

Areas of Application:Dialog/Deliberation

Dispute/Conflict Resolution

• Civic engagement/public dialogue• International & inter-group conflict• Labor/management, consumer disputes

alternative dispute resolution• Interpersonal disputes / mediation • Deliberative decision making (school,

work, home)

9

Social Deliberative Skills:Social/Emotional/Reflective

• 1. Social perspective taking (cognitive empathy, reciprocal role taking...)

• 2. Social perspective seeking (social inquiry, question asking skills...)

• 3. Social perspective monitoring (self-reflection, meta-dialogue...)

• 4. Social perspective weighing (reflective reasoning; comparing and contrasting views...)

10

Text Coding Scheme

Social Deliberative Skill:application of HOSs to me/you/we

Higher Order Skills • argumentation• critical thinking• explanation & clarification• inquiry/curiosity (question asking & investigation)• reflective judgment• meta-cognition• epistemic reasoning

Apply these skills, not to EXTERNAL REALITY (“IT”/problem domain) but to theINTERSUBJECTIVE domain

Higher Order Skills applied to:

SELFgoals; level of certainty; feelings, values, assumptions…

YOU goals, assumptions, feelings, values; perspective taking; "believing" & cognitive empathy…

WEagreements, goals; quality of the discourse/collaboration; differences and similarities in values, beliefs, goals, power, roles…

Corpora and Rater Agreement

13

Examples of Social Deliberative Skills/Behavior

From authentic dialogues in our online corpora

“ I am probably extremely biased because I am under 21 years old and in college. I wonder if as a 45 year old I will feel differently. ” (self reflection)

“I can’t help but imagine what that is like, for her and for her family.” (perspective taking)

Support/Scaffolding (vs. “Education”)

Online Dialogue &DELIBERATION

Outcomes:- Agreements/solutions

- Relationship, Trust (social capital)- SKILL USE (and practice)

Existing

Skills

Adaptive Support(4th party)

Passive Support(interface)

FacilitatorSupport

(Dashboard)

2. Classroom Studies

16

17

Mediem

Opinion Sliders

Study: Classroom Dialog• 26 College Students in 2 week

online discussion• 2 Topics: Trayvon Martin Shooting

& Gun Control• 3 Experimental conditions/ 3

discussion groups• 829 text segments from 369 posts• 43% of the segments coded as

"deliberate skill”

19

Experimental ConditionsExp Group N Gender Grade

Vanilla 8 (5 Female, 3 Male)

4 soph, 4 juniors, 0 seniors

Reflective Tools 8 (5 Female, 3 Male)

4 soph, 2 juniors, 2 seniors

(Sliders) 8 (Group omitted due to interaction issues)

• Sliders group omitted (did not use tools; poor group dynamics)

• V&R groups: 241 posts and 516 segments (average of 15.06 (SD = 7.45) posts/student)

• Mean words/post = 54 (SD = 42); mean characters/post = 299 (SD = 242)

20

Total Skill score adds:

• Intersubjectivity: perspective taking or question asking

• Meta-dialogue, discussing the quality of the dialogue• Meta-Topic: Birds eye or systemic view of the topic• Appreciation (Gratitude, affirmation of another's

idea or situation)• Source Reference (Mentioning a source, with a

reference or description; without a fact)• Apology

21

Total Skill vs. Condition

22

Intersub vs. Condition

23

Main Effect Exp. Group Total_

SD_SkillIntersubjectivespeech acts

Vanilla (N = 8) 0.29 (0.07) 0.20 (0.09)

Reflective Tools (N = 8) 0.40 (0.08) 0.30 (0.08)

• A significant difference and main effect between Total-SD-Score and grouping, F(1, 14) = 6.89, p = 0.02*, d = 1.46 (a large effect) in favor of the Reflective Tools group

• A significant relationship between Intersub and grouping, F(1, 14) = 4.81, p = 0.05*, d = 1.05 (a large effect) in favor of the Reflective Tools group

24

Comparison of Discussion Topics

Reflective tool use vs. topic

Story words vs. topic

Other Results

• No effects of sub-skill vs. gender, except females scored higher on Appreciation

• Positive correlation between Total Skill and post-survey scores on self-scored Engagement (r = 0.44) and Learning (r = 0.21) (no correlation vs. Enjoyment question)

• No correlation between Replies-from and Replied-to vs. experimental group

• The main effect of Condition vs. Total-skill came from the Trayvon discussion (Gun Control topic had less engagement)

• Most of main effect of Total-skill from the Intersub sub-skill

3. Facilitator Dashboard

27

28

View by Gender

29

Linguistic Features – LIWC

80+ features

5 categories

Linguistic process (e.g., total words per sentence, %

of pronouns)

Psychological process (e.g., affect,

cognition)

Paralinguistic dimensions (e.g.,

assents, fillers)

Punctuation (e.g., quotation marks,

exclamation marks)

Contents (excluded from this study)

30

Dashboard Text Tagging

31

Advice Screen

Settings

34

Next: LinkedRepresentations

Future: Additional MetricsCommon problems encountered in online facilitation

• Low or no participation of individuals or groups, or silences or lulls on the part of individuals, the entire group, or sub-groups

• Conversation domination by an individual or group • Inappropriate or disrespectful behavior • Off-topic conversation • Tension-filled disagreements, or high emotional content• Too much agreement or politeness • Misunderstanding due to missing communication skills

normally available in face-to-face communication

4. Automated Text Analysis/Classification

Text Analysis of Deliberative Skill: Using L1 Regularized Logistic Regression

with Psycholinguistic Features

37

Research Approach

• Analyze online dialogues through a variety of lexical, discourse, and gender demographic features

• Create machine learning classifiers to recognize social deliberative skills– “Total Skill”– Individual Skills (future work)

38

Text Coding Scheme

39

Total Skill score adds:

• Intersubjectivity: perspective taking or question asking

• Meta-dialogue, discussing the quality of the dialogue• Meta-Topic: Birds eye or systemic view of the topic• Appreciation (Gratitude, affirmation of another's

idea or situation)• Source Reference (Mentioning a source, with a

reference or description; without a fact)• Apology

Corpora and Rater Agreement

41

Demographic Features – Gender

• Data distribution

Motivation: Woolley et. al, have shown that women score higher on social sensitivity than men do.

42

Code Frequencies (classroom)  Inter-

subMeta_Dialogue

Meta_Topic

Apology Appreciation

Fact_Source

Source_Ref

#students 22 5 15 1 8 1 4

%segmts 25% 0.9% 5.5% 0.2% 1.3% 0.3% 1.2%

43

Code Frequencies in Several Domains

 Exp. Group Total_SD_Skill

Intersubjectivespeech acts

Vanilla (N = 8) 0.29 (0.07) 0.20 (0.09)

Reflective Tools (N = 8) 0.40 (0.08) 0.30 (0.08)

• A significant difference and main effect between Total-SD-Score and grouping, F(1, 14) = 6.89, p = 0.02*, d = 1.46 (a large effect) in favor of the Reflective Tools group

• A significant relationship between Intersub and grouping, F(1, 14) = 4.81, p = 0.05*, d = 1.05 (a large effect) in favor of the Reflective Tools group

44

Linguistic Features – LIWC

80+ features

5 categories

Linguistic process (e.g.,

total words per sentence, % of

pronouns)

Psychological process

(e.g., affect, cognition)

Paralinguistic dimensions

(e.g., assents, fillers)

Punctuation (e.g., quotation

marks, exclamation

marks)

Contents (excluded from this

study)

45

Discourse Features – Coh-Metrix

100+ features

8 categories

Narrativity Referenti

al cohesion

Syntactic simplicity

Word concrete

nessCausal cohesion

Verb cohesion

Logical cohesion

Temporal cohesion

46

Machine Learning Method

• L1 Regularized Logistic Regression

–Auto-select features while learning

–High generalizability via minimizing training loss and selecting a sparse model

–High transparency like a “glass-box” model

47

Performance Metrics

• Accuracy What percent of all predictions were correct?

Precision What percent of the positive predictions were correct?

• Recall What percent of the positive cases were caught?

• F2

Weighted average of precision and recall that weights

recall twice as high

Study 1. Single Classroom Dialog

• 26 College Students in 2 week online discussion

• 3 small discussion groups (of 8 or 9)• 2 Topics: Trayvon Martin Shooting & Gun

Control• 829 text segments from 369 posts• 43% of the segments coded as "deliberate

skill”

Predictive performance (in %) of L1 regularized logistic regression

built using different type of features

Results

• Moderate Recall (68%) and F2 (65%)• LIWC features outperformed Coh-Metrix • Adding gender and grade level features did

not improve performance• Possibly encoded within LIWC/Coh-Metrix

51

Study 2: Multi-Domain Dialogue Analysis

• College dialogues – 4 college classes– Posts from college students from a variety of

disciplines participating in e-discussions, about controversial topics.

• Civic deliberation– E-Democracy.org, neighborhood discussion about

ethnic tensions in a multi-racial community.

• Professional community negotiation– Email exchanges among faculty of two academic

communities deciding where to schedule a meeting

52

Preliminary ML methods comparison

Conclusion:• L1-LRL slightly

outperformed Naive Bayes and SVM

53

Study 2 Experimental Design(using L1-RLR)

• Goals– Study feature effects on prediction

performance of machine learning models

• Design of 2 scenarios and 54 experiments

– In-domain analysis (6*3 evaluations)

• Evaluate six possible feature configurations in each domain

– Cross-domain analysis (6*6 evaluations)

• Evaluate six possible feature configurations in six domain pairs (training and testing)

54

55

In-Domain Training

In-domain training: Results

• Gender did not predict; nor did adding it to other features improve prediction

• Classroom domain had poor performance (probably due to data skew)

• LIWC performed better in Faculty dialogue recall 90%)

• Coh-Metrix performed better in Civic dialogue (recall 84%)

Data Skew per Domain

Cross-Domain Training

Cross-Domain Training Results

• College domain prediction much better when training in other domains

• Faculty domain best for training overall (Recall: 89% Civic; 87% College; 90% Faculty)

• In general LIWC features do slightly better than CohMetrix, and combining them does not improve performance

60

Conclusions

62

Future Work-Text Analysis

Create multi-task machine

learning models with advanced

regularizes (e.g., sparse group

Lasso) to simultaneously identify

each component social

deliberative skills from online

communication

www.SocialDeliberativeSkills.com

extra slides

Top 10 LIWC features learned by L1 regularized logistic regression

Total Skill component correlations