Toward Defining, Justifying, Measuring, and Supporting

Social Deliberative Skills

Tom MurrayUMass Amherst

At AIED July 2013, Memphis: Workshop on Self-Regulated Learning in Educational

Technologies: Supporting, modeling, evaluating, and fostering metacognition with computer-based learning environment

Group & CollaborativeWork/Leanin

g

Conflict Resolution

Meaning Negotiation

Problem solving

Planning

Brainstorming & Creativity

Inquiry

Decision making

Knowledge building

Group dynamics

(form, storm, norm)

Peer help/tutoring

3

Social Deliberative Skills:Social/Emotional/Reflective

• 1. Social perspective taking (cognitive empathy, reciprocal role taking...)

• 2. Social perspective seeking (social inquiry, question asking skills...)

• 3. Social perspective monitoring (self-reflection, meta-dialogue...)

• 4. Social perspective weighing (reflective reasoning; comparing and contrasting views...)

Social Deliberative

Skills

Perspective Taking

Social Metacognition

Reflective ReasoningEmpathy

Dialogue and Deliberation

skills

Meaning NegotiationConflict Resolution

Skills needed to bridge different perspectives to build mutual understanding and mutual regard

Social Deliberative Skills: The capacity to deal productively with heterogeneous goals, values, or perspectives in dialogue and deliberation

– Including: collaboration, problem solving, knowledge building, inquiry learning...

Overview

• Background• Supporting them online

– Participants – Facilitators

• Measuring them – Human coding– Machine classification

• Deeper exploration of their meaning– And issues with construct definitions (ontology)

1. Background

8

Text Coding Scheme

Corpora and Rater Agreement

10

Examples of Social Deliberative Skills/Behavior

From authentic dialogues in our online corpora

“ I am probably extremely biased because I am under 21 years old and in college. I wonder if as a 45 year old I will feel differently. ” (self reflection)

“I can’t help but imagine what that is like, for her and for her family.” (perspective taking)

11

Code Frequencies in Several Domains

 Exp. Group Total_SD_Skill

Intersubjectivespeech acts

Vanilla (N = 8) 0.29 (0.07) 0.20 (0.09)

Reflective Tools (N = 8) 0.40 (0.08) 0.30 (0.08)

• A significant difference and main effect between Total-SD-Score and grouping, F(1, 14) = 6.89, p = 0.02*, d = 1.46 (a large effect) in favor of the Reflective Tools group

• A significant relationship between Intersub and grouping, F(1, 14) = 4.81, p = 0.05*, d = 1.05 (a large effect) in favor of the Reflective Tools group

Support/Scaffolding (vs. “Education”)

Online Dialogue &DELIBERATION

Outcomes:- Agreements/solutions

- Relationship, Trust (social capital)- SKILL USE (and practice)

Existing

Skills

Adaptive Support(4th party)

Passive Support(interface)

FacilitatorSupport

(Dashboard)

13

14

Mediem

Opinion Sliders

15

Graduate Class 2012—Total Skill

16

17

CohMetrixdiscourse & coherence

LIWClexical categories

18

19

feature comparisons

In-domain Cross-domainTraining corpus

Civic deliberation

Professional community negotiation

Civic deliberation

Professional community negotiation

  Testing corpus

Civic deliberation

Professional community negotiation

Professional community negotiation

Civic deliberation

Gender

Accuracy 56.8 52.7 52.7 56.8Precision 56.8 52.7 52.7 56.8

Recall 100.0 100.0 100.0 100.0F2 86.8 84.8 84.8 86.8

LIWC

Accuracy 55.3 55.3 56.2 59.6Precision 58.9 54.6 56.9 59.6

Recall 70.7 90.0 69.7 89.3F2 68.0 79.7 66.7 81.2

Cohmetrix

Accuracy 62.1 54.8 53.4 53.3Precision 61.4 55.3 53.5 58.3

Recall 83.6 74.0 90.0 62.2F2 77.9 69.3 79.2 61.4

LIWC+Gender

Accuracy 52.3 54.6 56.2 60.9Precision 56.8 54.2 56.8 60.6

Recall 67.1 89.6 70.6 88.9F2 64.8 79.3 67.3 81.3

Cohmetrix +Gender

Accuracy 62.4 55.5 53.9 53.8Precision 62.7 55.6 53.9 58.5

Recall 83.6 77.5 87.0 64.0F2 78.3 71.8 77.5 62.8

LIWC +Cohmetrix

Accuracy 62.4 54.5 54.1 55.6Precision 63.3 55.1 54.0 60.0

Recall 84.4 74.5 87.0 65.3F2 79.2 69.6 77.5 64.2

All

Accuracy 62.1 54.3 52.7 54.8Precision 62.3 55.0 53.1 59.7

Recall 84.4 74.0 88.7 63.1F2 78.8 69.2 78.2 62.4

Exploring Social Deliberative Skills

Social Deliberative Skills:Literature BACKGROUND

• Dialogue Acts, Argumentation• Reflective & Critical Reasoning• Meta-skills• Civic Deliberation• Accountable Talk• Research into Perspective taking and Empathy

Dialogue Act Schemes• M. Baker (Rainbow, 2007)

– Social relation – Interaction management – Task management– Opinions – Argumentation – Broaden and deepen

• Dejong (Tr-Diagram, 2005) – Argument– Explanation– Challenge (but)– Evidence– Support/Refute

• Kim (2007)– Acknowledge/

Support/Complement– Inform, Command,

Announce– Answer/Suggestion/

Advice– Correction/Objection– Elaboration/Description– Question

Reflective Reasoning, Critical Thinking, Argumentation

• Reflective Reasoning– Ill-structured problems – about which reasonable people reasonably

disagree• Critical Thinking

– analyze– interpret– integrate

D. Kuhn: Ill-defined constructs in Higher Order Thinking

Argumentation

Inquiry

Critical thinking

Metacognition

Reflective Judgment

Epistemological Knowledge

Metacognition(Winne; Azedevo; Wellman)

• Phases: planning, monitoring, control/regulation, reflection/assessment (repeat)

• Domains: cognitive, motivational/affective, behavioral, contextual; memory, attention; problem solving state

• Self-reflection; self-explanation; help/info-seeking and SR learning

Social Metacognition(White; Scardemalia; Collins)

• Meta-knowledge/skill about:– Self expertise, incl. collab. and communic. skill– Roles & relationships– Social context (and mediation)– Knowledge-building: monitoring; improving; rise-

aboves

Civic Deliberation

– Dahlberg (2001) (& Habermas): – reflexivity, ideal role taking, sincerity, inclusion – exchange of criticizable moral-practical validity

claims

Analytic Aspects of Deliberation:Stromer-Galley

• Introduction - welcoming, context, etc.• Deliberation Process • Technical Process• Meta-Talk (consensus, conflict, clarity)• Summary• Disagree/Agree• Off-topic move• Reasoned Opinion (fact, question, elab. opinion)• Sourcing• Intervention • Invite Others• Social (praise, apology, off topic)

29

Analytic/Cognitive vs. Social/EmotionalGastil & Black: Key features of deliberative dialogue

Accountable Talk

• Social: Listen (pay attention to…), Summarize, Build upon, Refer to

• Knowledge: Verify (check), Unpack (explain), Support (examples, evidence), Link

• Reason: Defend, Challenge, Combine, Predict

Social Deliberative Skills – v1

Primary SD-Skills

• Differentiating facts/opinions (knowing how to reason about each)

• Reflecting on biases and assumptions (mostly one's own but also others)

• Perspective taking (of actual interlocutors; of other groups/identities/cultures, etc.)

• Reflecting on the dialog as a whole (meta-dialogue)

33

Social Deliberative Skills V-2Social/Emotional/Reflective

• 1. Social perspective taking (cognitive empathy, reciprocal role taking...)

• 2. Social perspective seeking (social inquiry, question asking skills...)

• 3. Social perspective monitoring (self-reflection, meta-dialogue...)

• 4. Social perspective weighing (reflective reasoning; comparing and contrasting views...)

34

Text Coding Scheme

Social Deliberative Skill:application of HOSs to me/you/we

Higher Order Skills • argumentation• critical thinking• explanation & clarification• inquiry/curiosity (questioning)• reflective judgment• meta-cognition• epistemic reasoning

Apply these skills, not to EXTERNAL REALITY (“IT”/problem domain) but to theINTERSUBJECTIVE domain

Higher Order Skills applied to:

SELFgoals; level of certainty; feelings, values, assumptions…

YOU goals, assumptions, feelings, values; perspective taking; "believing" & cognitive empathy…

WEagreements, goals; quality of the discourse/collaboration; differences and similarities in values, beliefs, goals, power, roles…

Next: Support & Interventions

• “Teaching:” role-play, simulated dialogues...• Research on Perspective taking & Empathy?• Computer-based

– Scripts– Sentence Openers– Reification (& passive reminders)– Awareness Tools

www.SocialDeliberativeSkills.comProject Collaborators: Bev Woolf, Xiaoxi Xu, Lynn Stephens, Leah Wing, Natasha ShrikantAnd thanks to Art and Nia from CohMetrics lab.

38

Project Overview

A journey through ontological conundrums

• Below is additional material related to the unavoidability of construct overlap and indeterminacy in defining and using abstract concepts such as metacognition, inquiry, etc. (See overlap in Kuhn slide above)

“ ...to Understand, Measure & Support SD-skills”

• Reviewer: “How can study them if you don’t understand them and can’t define them precisely?”

• Scholarly work is "notoriously fraught with definitional disagreement" (Shermer, 2011)

An invitation to co-explore Ontological Indeterminacy

Kurt Fischer’s Dynamic Skill Theory

• Skills develop in response to real tasks• Higher order skills are built doing complex

tasks; no isolated task, therefore no isolated skill (compare “leadership” skill with “territoriality” biological drive)

• ...skills [and knowledge] are not isolated units, but rather

function together in complex structures of inter participation...an ecosystem ...any given skill requires the existence of various others as component [or

interacting] parts...

Therefore....

• Definition of task takes precedence over definition of the skill

Example: “intelligence” and IQ

• Real life target Task• Skills (& knowledge)• Measurement

Caution: the construct definition can be reduced to the measurement definition or enactment

SD-Skills: task-oriented definition

Skills needed to bridge different perspectives to build mutual understanding and mutual regard

Social Deliberative Skills: The capacity to deal productively with heterogeneous goals, values, or perspectives in dialogue and deliberation

– Including: collaboration, problem solving, knowledge building, inquiry learning...

Concept Indeterminacy(Lakoff)

• Abstract Concepts:– Graded (fuzzy)– Have “metaphorical pluralism”– Metaphorical (limited by embodiment)

• More abstract => More indeterminate

Tom Murray | www.perspegrity.com | August 2010 46

Two sources of indeterminacy

• 1. Task structure: Higher order skills are built doing complex tasks; no isolated task therefore no isolated skill

• 2. Nature of categories: Abstract concepts are indeterminate by nature

Real phenomena tend to be chaotic, organic; not categorized

extra slides

Stromer-GalleySD-Skills Strommer-Galley

WELCOMING Introduction - welcoming, context, etc.

PROC-EXPL - process-explanation Deliberation Process

TECH Technical Process

META_SUM Summarize

AGREE Disagree/Agree

DISAGREE Disagree/Agree

OFF_TOPIC Off-topic move

MEDIATE Intervention || Invite Others

SOCIAL Social

Code-descriptions and frequencies

Perspective taking (15%), Big picture thinking (8%), meta-dialogue (5%), self-reflection (3%), Arg_gen (~30%)

Debate, Dialogue and DeliberationDeliberation: “thoughtful, careful, or lengthy consideration by individuals; and formal discussion and debate in groups” (Davies & Chandler 2011)