Polyphonic model, analysis method, software and sonification of discourse

Ştefan Trăuşan-Matu

University Politehnica of Bucharest

Computer Science Department

Polyphonic Analysis of Discourse in Texts and

in Collaborative Learning Chats

Contents

The Polyphonic Model

Polyphonic Analysis

Implementations of the Polyphonic Analysis

The PolyCAFe Analysis System

Other Applications

Conclusions

12/14/2013 2 Stefan Trausan-Matu

Polyphony- An Unitary Model of

Human Communication

Mediation:

Using natural language (words) in

texts

hypertexts

discussion forums

conversations

but also

Non verbal communication (e.g.

gestures)



Human Group Communication

Considering rather a dialogistic, post-

structuralist (Bakhtin, Kristeva) than a

mechanistic perspective on communication

Rather a socio-cultural (Vygotsky) approach

than a cognitivist one (like in Artificial

Intelligence) but considering the both

Ethnomethodology (Garfinkel), Conversation

Analysis (Sacks, Schegloff, Jefferson)



Group Communication


Applicable to:

Small groups (e.g. virtual teams

collaborating by chat or forums – the

INTER-ANIMATION phenomenon

appears)

Large groups – social networks

Global level - intertextuality


Group Communication and

Intertextuality

It may be used in IT implementations

using

Natural language Processing

Machine learning

Social Network analysis

Specific techniques (inter-animation

and collaboration analysis)


The Polyphonic Model


Polyphony

Appears in music (e.g. J.S.Bach) and in texts (Bakhtin)

The Polyphonic

Model (Trausan-Matu, Handbook of Hybrid Learning, 2010)

Analysis method (Trausan-Matu and Rebedea, 2010)

Computer support tools for the polyphonic analysis of F2F, online and offline conversations: The “Polyphony” system (Trausan-Matu and all, 2007)

ASAP (Dascalu, Chioasca and Trausan-Matu, 2008)

PolyCAFe (Trausan-Matu, Rebedea and Dascalu, 2011; Rebedea, Dascalu, Trausan-Matu and all, 2010)

Collaboration regions detection (Banica, Trausan-Matu and Rebedea, 2011)

Detection of the Important moments (Chiru and Trausan-Matu, 2012)


Polyphony

A group of participants that, each of them

keeps their individuality, personality, creativity,

but also collaborate to achieve a common

goal, trying to solve dissonances

A merge of:

Unity vs. Difference

Melody (longitudinal) and Harmony (transversal)

Dissonance – Consonance Cycles –

centrifugal/centripetal forces

Inter-animation of voices – inter-animation

patterns 12/14/2013 9 Stefan Trausan-Matu

The Polyphonic Model Polyphony = Model of collaboration and interaction

(Trausan-Matu, Stahl and Zemel, 2005)

Human communication in knowledge construction and collaboration are processes in which words and other utterances are linked in parallel threads which interact similarly to voices in polyphonic music

Repetition and rhythm are essential


Polyphony


Dialogism and Polyphony (Bakhtin) Mikhail Bakhtin:

• Utterances (not sentences) should be the unit of analysis

• “These are different voices singing variously on a single theme. This is indeed 'multivoicedness,' exposing the diversity of life and the great complexity of human experience. 'Everything in life is counterpoint, that is, opposition,' “ (Bakhtin, 1984)

• “… Any true understanding is dialogic in nature” (Voloshinov-Bakhtin, 1973)

• Speech genres

• Polyphony Inter-animation of voices

• Basis for the CSCL paradigm (Koschman, 1999)

• Opposed to de Saussure ideas: • Real life dialog should be the focus, not written text

• Words are not arbitrary


Bakhtin’s Polyphony

Everything is a dialog (applying not only to speech and text)

Utterances

Voices Inter-animations among voices


Utterances Utterances (not sentences, as in ‘classical’ linguistics) should

be the unit of analysis (Bakhtin) Utterances are acts An utterances may be a:

Word Turn, a reply in a conversation, chat or forum Sentence Text Image (picture, diagramatic representation, etc.) Gesture (individual or group) Thought – inner utterances – inner speech

Utterances should be considered at different granularities

Utterances are linked in threads formed by: Explicit links (VMT chat environment; forum’s replies) - uptakes

(Suthers, 2010) Implicit links, detected by Natural Language Processing techniques

– contingencies, uptakes (Suthers, 2010)

Utterances may become voices


Voices Distinctive presences in a group, influencing the

other voices

Generated by utterances (singular or repeated) Correspond to:

participants (may also be inner voices) groups of participants (e.g. collective or collaborative

utterances) chains or threads of words or concepts: repeated words lexical chains co-references reasoning or argumentation rhetorical schemas

Each utterance may contain multiple voices Voices continue and influence each other through

explicit or implicit links.


Inter-animation patterns

Longitudinal

Adjacency pairs

Repetitions

Elaboration

Convergence

Cumulative talk

Repair

Transversal, differential

Dissonance


The Polyphonic Method applications

Chat conversations with multiple participants for:

CSCL: K-12 students solving mathematics problems both individually and collaboratively in the VMT project at Drexel University, Philadelphia, US

CS students at University Politehnica of Bucharest , Romania at

o CHI course in Romanian and French – role playing and debate

o Natural Language Processing - role playing and debate

o Algorithm Design – problem solving

Fostering creativity – brainstorming, synectics

F2F collaborative learning (Suthers & all, 2011)

Analysis of Rhythm

Metacognition (conversation & essays)

OpenSimDeveloper dataset

Intertextuality 12/14/2013 18 Stefan Trausan-Matu

Computer Support for the

Polyphonic Analysis


LTfLL - EU FP7 Project (2008-2011) and

NSF Virtual Math Teams Project http://www.ltfll-project.org/ http://mathforum.org

Language Technologies for Lifelong Learning Netherlands, France, United Kingdom, Germany, Ausria, Romania,

Bulgaria

PolyCAFe system (Polyphony-based Collaboration

Analysis and Feedback generation)

The system has been validated with students and tutors in

University of Manchester, UK

Politehnica University of Bucharest, Romania


Chat-based CSCL

K-12 students solving mathematics problems both individually and collaboratively in the Virtual Math Teams (VMT) project at Drexel University, Philadelphia, US (Directed by Gerry Stahl)

Computer Science students at University Politehnica of Bucharest (UPB), Romania at Human-Computer Interaction course in Romanian and

French – role playing and debate Natural Language Processing - role playing and debate Algorithm Design – problem solving


Theories for analysing

multi-parties conversation

Discourse analysis (Tannen)

Conversation analysis (Sacks, Jefferson, Schegloff)

Accountable talk (Resnick)

Transactivity (Teasley, Berkowitz & Gibbs, Joshi &

Rose)

Events/contingencies, coordinations/uptakes

(Suthers)

Inter-animation (Bakhtin, Wegerif, Trausan-Matu)

Polyphony (Bakhtin, Trausan-Matu)


Analysis methods

TF-IDF

Latent Semantic Analysis

Naïve Bayes

Social Network Analysis

WordNet (wordnet.princeton.edu)

Support Vector Machines

Collin’s perceptron

TagHelper environment


Analyis methods TF-IDF

Latent Semantic Analysis Almost all are based also on

Naïve Bayes a two interlocutors

Social Network Analysis model, in which

WordNet (wordnet.princeton.edu) one person speaks

Support Vector Machines at a time, resulting

Collin’s perceptron one discussion thread

TagHelper environment


NLP pipe

spelling correction, stemmer, tokenizer, Named Entity

Recognizer, POS tagger and parser, and NP-chunker.

Stanford NLP software

(http://nlp.stanford.edu/software)

Spellchecker : Jazzy

http://www.ibm.com/developerworks/java/library/j-

jazzy/

Alternative NLP pipes are under development,

GATE (http://gate.ac.uk)

LingPipe (http://aliasi.com/lingpipe/).


Social network analysis

Consider explicit and implicit referencing as arcs

between participants, which are the nodes

A kind of page-rank algorithm – an utterance is

important if it is referred by important utterances;

The strength of a voice (of an utterance)

depends on the strength of the utterances that

refer to it

Determines if a person is central/peripheral


Polyphony, Inter-animation and

Collaboration analysis

Assign an importance value for each utterance

considering several indicators of inter-animation

(collaboration) Detection of voices (chains) inter-animation patterns

(Trausan-Matu) in the chat

Consider several criteria such as the presence in the

chat of questions, agreement, disagreement

Presence of others’ voices

Social Networks metrics

Machine learning approach (genetic algorithms and

neural networks) for tuning the 12/14/2013 29 Stefan Trausan-Matu

Computational details (Trausan-Matu, Dascalu and Dessus, ITS 2012;

Dascalu, Trausan-Matu and all, 2010, 2011))


Representations:

Conversation graph

For each participant there is a separate horizontal line in the representation

Each utterance is placed in the line corresponding to the issuer of that utterance, according to the emission time, alligned from left to right The explicit references among utterances are

depicted using connecting lines distinctively colored

The implicit references (deduced by the system) are represented using other color that the explicit ones

An estimation of the strength of each utterance (when available) is represented as a bar chart


Representations:

Weaving of Voices

Voices in the conversation graph

Participants = horizontal lines

Threads of repeated words or phrases = differently

colored threads


37 12/14/2013 37 Stefan Trausan-Matu

Validation (Rebedea, Dascalu, Trausan-Matu and all, ECTEL 2010;

Rebedea, Dascalu, Trausan-Matu and all, ECTEL 2011)


Other applications


Analysis Dimensions (types of voices)

in Face-to-Face Settings (Trausan-Matu, in Suthers & all, (eds.) 2013)

Spoken dialog

Body language

Individual

Collective

The visual dimension

Visual data on the blackboard

What others participants do

Others’ body language

Internal dialogue (at an intra-mental level)

Echoes


Inner utterances, inner speech

“Mead (1934) called thought a <<conversation

with the generalized other,>> implying that when

we think individually we attempt to respond-

internally and vicariously-to the imagined

responses of others to our ideas and arguments.”

(Resnick & all, 1993)

“There are no ontological differences between

inner and outer speech” (Clark and Holquist,

1984).


Detecting important moments (Chiru and Trausan-Matu, ITS 2012)


Graphical representation of topic’s

rhythmicity (Chiru, Cojocaru, Trausan-Matu and Rebedea, ISMIS 2011)

High

rhythmicity for

all topics –

these were

debated in

parallel as it

can be seen by

the lack of flat

lines near the

left side of the

representation.

Low rhythmicity

– flat lines on

the left side of

the graphic

showing that

the topic that

they represent

has not been

debated in

those parts of

the chat. 12/14/2013 47 Stefan Trausan-Matu

Determination of collaborative

regions (Banica, Trausan-Matu and Rebedea, CSCL 2011)


Metacognition (Trausan-Matu, Dascalu and Dessus, ITS 2012)

Combining Chats and Texts

Intertextuality


ReaderBench (Dascalu, Trausan-Matu, Dessus, 2012)


Creativity fostering

(e.g. brainstorming)(Trausan-Matu, 2011)


Opinion mining (Musat, Velcin, Rizoiu, and Trausan-Matu,

2011)


Topic Modeling (Musat and Trausan-Matu, 2011)

No generally accepted definition for a “topic” Document clusters Abstractions based on document clusters

Labels; Centroids, etc

(Word, Probability) pairs

Bayesian statistical models Topics – distributions over words Documents – distributions over topics Generative model Topic Intertwining Conceptually similar to the ideas of Mikhail Bakhtin

Topics and voices


Topic Modeling (Musat and Trausan-Matu, 2011)

LDA/pLSA/hLDA/CTM

Each newer version corrects some flaws of the earlier

ones

However the traditional means of testing the accuracy

have been proven wrong

Even more reason to look into the problem of evaluating

the models

LDA

Readily available

Mallet

Easily reproducible experiments

Well known topic model;


Intertextuality analysis (Ghiban & Trausan-Matu, 2012)

Voice I

Voice II

Voice III

Voice I

Voice II

Voice III

In dialog

Text 1 Text 2 Text 3

Text 4

Text 1

Text 2

Text 3

In dialog in text 4


Intertextuality analysis (Ghiban & Trausan-Matu, 2012)

Theme 2 and Theme

3 may have the

same words but only

different concepts

Section 1 and 6 are dialogical or

polyphonical. They may present a

higher force of expresivity. 12/14/2013 56 Stefan Trausan-Matu

Analysis of interethnic discourse

(Trausan-Matu, 2012)

Needed a corpus of texts with time stamps

Extract recurrent concepts in texts

Identify historical events

Generate time series

Analysis of correlations between time series

Analysis of the polyphonic structure


Time Series Analysis of News (Badea & Trausan-Matu, 2013)


Music Composition at K-Teams Laboratory (Master and Bachelor Thesis coordinated by Prof. Trausan-Matu)

Genetic Algorithms

Celular automata

Artificial chemistry

Constraint-based systems

Accompaniments generation with Markov Models

Random generation

Automatic counterpoint generation according to Fux rules

Chat sonification


Chat sonification (Stefan Trausan-Matu and Alexandru Calinescu)


Chat sonification


Chat sonification

(Orchestration by Serban Nichifor)


Conclusions

The polyphonic model may apply to non-verbal

collaboration and intra-subjective (inner thinking)

as well as inter-subjective levels

A combination of Conversational Analysis with

Natural Language Processing is possible

(cognitive and socio-cultural)

Learning analytics tools that combine the two

perspectives and the Polyphonic Model may be

developed


Thank you!

[email protected]

http://www.racai.ro/~trausan

Questions?


mailto:[email protected]

http://www.racai.ro/~trausan

Polyphonic model, analysis method, software and sonification of discourse

Technology

Transcript of Polyphonic model, analysis method, software and sonification of discourse