Stefan Trausan-Matu

PolyCAFe and Social Learning

Support for CSCL in LTfLL

9th December 2010

Stefan Trausan-Matu

Traian Rebedea

Mihai Dascalu

Vlad Posea

“Politehnica" University of Bucharest

Computer Science Department

General dataGeneral data

9 December 20102

The LTfLL - EU FP7 STREP Project (2008-2011)

• Open Universiteit Nederland (coordinator)

• The University of Manchester

• Open University UK

• Universiteit Utrecht

• Eberhard Karls Universität Tübingen• Eberhard Karls Universität Tübingen

• Wirtschaftsuniversität Wien

• Université Pierre-Mendès France, Grenoble

• Politehnica University of Bucharest (PUB)

• Institute for parallel processing of the Bulgarian Academy

• Aurus Kennis- en Trainingssystemen BV

• BIT MEDIA E-learning solution GMBH and CO KG

9 December 20103

Past International Research Projects

Related to LTfLL

� The cognitive paradigm (e.g. Web semantic)� Portable AI Lab (PAIL) – IDSIA Lugano (1994-1995)� PeKADS - EU Copernicus (1994-1996)� LARFLAST – EU Copernicus (1998-2001)� IKF – EU EUREKA (2002-2003)� Towntology – EU COST Action (2005-2008)

� The socio-cultural paradigm (e.g. Web2.0)� VMT –USA NSF (2005-2007)� EU-NCIT – EU FP6 (2005-2008)� Cooper – EU FP6 (2005-2007)

9 December 20104

The Outcomes of the LTfLL Project

Prototypes of next-generation services built on

advanced research on the application of

language technologies in education.

9 December 20105

http://www.ltfll-project.org

LTfLL Services

9 December 20106

LTfLL Development Cycles

9 December 20107

Workpackages

OUNL 19 7 10 11 11 3 2 63

UU 3 5 5 2 18 7 3 1 44

UTU 7 6 18 6 3 1 41

WUW 10 3 4 7 4 3 1 32

UPMF 1 1 19 3 2 1 27

PUB-NCIT 1 3 19 17 8 3 1 52

AURUS

KTS

3 6 1 6 3 1 20

Work

packageWP1 WP2 WP3 WP4 WP5 WP6 WP7 WP8 WP9 TOTAL per

Beneficiary

9 December 20108

KTS

UNIMAN 5 12 6 14 3 1 41

IPP-BAS 3 6 11 15 9 3 1 48

BIT

MEDIA

15 3 5 6 1 30

OU 8 3 6 3 1 1 22

TOTAL 19 58 51 68 53 56 70 33 12 420

PUB Implementation Team� Prof.dr.ing. Stefan Trausan-Matu

� Prof.dr.ing. Valentin Cristea

� As.drd.ing. Traian Rebedea

� As.drd.ing. Vlad Posea

� As.drd.ing. Mihai Dascalu

� As.drd.ing. Costin Chiru

� As.drd.ing. Dan Mihaila

� Ing. Alexandru Gartner

� Ing. Erol Chioasca

Students which helped at the implementation:

� Dan Banica

� Mihai Nicolae

� Iulia Pasov

� Ionela Voinescu

� Iulia Moscalenco

� Oana Mihai

� Alexandru Georgescu

9 December 20109

PUB Publications on LTfLL� 29 papers, 12 in Proceedings ISI, 23 in

international databases,

� 2 NLPSL Workshops with Proceedings (Eds. S. Trausan-Matu, P. Dessus)

� 4 book chapters (including published at Springer � 4 book chapters (including published at Springer and Hershey)

9 December 201010

The Problems Solved by

PolyCAFe(a Polyphony-Based System for Collaboration (a Polyphony-Based System for Collaboration

Analysis and Feedback Generation)

9 December 201011

Chat Conversations with Multiple

Participants

� Multiple participants (≥3), conferencing style

� Particular features – multiple, parallel discussion

chains !!!

There is a need for� There is a need for

� Determining important utterances

� Contributions of the participants

� Degree of collaboration - inter-animation analysis

9 December 201012

Example: CSCL assignment� Students had to debate in chat sessions in

groups ranging from 3 to 8

� In the first part of the conversation, each student had to defend a technology by presenting its features and advantages and criticize the others features and advantages and criticize the others by invoking their flaws and drawbacks

� In the final part of the chat, they had to discuss on how they could integrate all these technologies in a single online collaboration platform

9 December 201013

CSCL assignment: Problems

�How to assist teachers in evaluating students’ work in chats?

�Offer assistance to students

�Abstraction tools�Abstraction tools

�Automatic feedback

9 December 201014

Experiments with Chat-based

CSCL

�K-12 students solving mathematics problems both individually and collaboratively in the VMT project at Drexel University, Philadelphia, USA

�Computer Science students at Bucharest “Politehnica” �Computer Science students at Bucharest “Politehnica” University, Romania at

�Human-Computer Interaction course in Romanian and French –role playing and debate�Natural Language Processing - role playing and debate�Algorithm Design – problem solving

9 December 201015

The VMT chat environment

9 December 201016

VMT Referencing facility

9 December 201017

Theoretical Basis of Theoretical Basis of

PolyCAFe

9 December 201018

Paradigms about KnowledgeCognitive Socio-cultural

Newell, Simon Vygotsky, Bakhtin

“Knowledge is in the head” “Knowledge is in the community”

Artificial Intelligence,

Natural Language Processing

Theory of Activity,

Collaborative systemsNatural Language Processing Collaborative systems

Ontologies Folksonomies

Semantic Web Social Web (Web2.0)

Intelligent Tutoring Systems Computer-Supported Collaborative Learning

9 December 201019

Computer Supported Collaborative

Learning

� A new paradigm in learning with computers (Koshmann, 1999):

� Knowledge is constructed socially (Vygotsky)� Knowledge is constructed socially (Vygotsky)� Induced by the spread of forums, chats, blogs,

wikis and folksonomies � learning in (on-line) virtual teams and/or communities

9 December 201020

Dialogism – Mikhail Bakhtin

• Basis for the CSCL paradigm (Koschman, 1999)

• “… Any true understanding is dialogic in nature” (Voloshinov-Bakhtin, 1973)nature” (Voloshinov-Bakhtin, 1973)

• Opposed to de Saussure ideas, which are the basis for Natural Language Processing

� Polyphony

� Inter-animation of voices

9 December 201021

The Polyphonic Model of CSCL(Trausan-Matu, Stahl and Zemel, 2005, http://mathforum.org/wikis/uploads/Stefan_Interanimation.doc)

� A polyphony of voices characterizes any linguistic phenomenon (Bakhtin) including CSCL chats

� Inter-animation (Bakhtin, Wegerif) may be detected in interactions and it may be used for analyzing collaboration and assessing learnersassessing learners

� Integrating NLP techniques with polyphony identification and Social Network Analysis may provide a way for analyzing the contributions of each participant and their collaboration.

� Inter-animation and polyphony appears also in non-verbal interactions

� Consider threads (voices which last) rather than analyzing pairs of utterances

9 December 201022

Polyphonic analysis(Trausan-Matu & Stahl, 2007, http://gerrystahl.net/vmtwiki/stefan.pdf)

9 December 201023

249 December 201024

Words, voices and threads� Different positions assigned to participants – different

voices

� Additional voices – frequent concepts – repeated

words become voices, stronger or weaker

� Voices continue and influence each other through

explicit or implicit links. explicit or implicit links.

� Voices correspond to chains or threads of utterances:� repeated words

� lexical chains

� co-references

� reasoning or argumentation

� rhetorical schemas

9 December 201025

Analysis Units in the Polyphonic

Model

� Words

� Utterances

� Pairs of utterances (links)

� Threads� Threads

� Voices

� Participants

9 December 201026

Units of Interaction

� Echoes of voices

� Polyphonic-contrapuntal weaving

� Inter-animation

Links between utterances and between words� Links between utterances and between words

� Links may be:

� implicit

� explicit

9 December 201027

Inter-animation Patterns(Trausan-Matu, Stahl & Sarmiento, 2007)

� Longitudinal� Adjacency pairs

� Repetitions

� Elaboration

� Cumulative talk (collaborative utterances)� Cumulative talk (collaborative utterances)

� Repair

� Transversal � Convergence

� Differential

� Dissonance

9 December 201028

Implementation Details of PolyCAFeImplementation Details of PolyCAFe

9 December 201029

Architecture of PolyCAFE

309 December 201030

9 December 201031

NLP pipe� spelling correction, stemmer, tokenizer, Named Entity

Recognizer, POS tagger and parser, and NP-chunker.

Stanford NLP software

(http://nlp.stanford.edu/software)

� Spellchecker : Jazzy

http://www.ibm.com/developerworks/java/library/j-http://www.ibm.com/developerworks/java/library/j-

jazzy/

� Alternative NLP pipes are under development,

� GATE (http://gate.ac.uk)

� LingPipe (http://aliasi.com/lingpipe/).

9 December 201032

Functional architecture

Surface Analysis

Readability & Page essay grading

Surface Analysis

Readability & Page essay grading

Morphological Analysis and POS Tagging

Morphological Analysis and POS Tagging

Semantic Evaluation

(LSA)

Semantic Evaluation

(LSA)

Social Network

Analysis

Grading Process

9 December 201033

Tagged LSA• Corpus of chats focused on collaborative

technologies

• POS Tagging

• Stemming

• Segmentation• Segmentation

– Participants

– Fixed Window

• Cosine similarity

Term-Doc Matrix + Tf – Idf

9 December 201034

Vector space visualization

Radial Model

Physical Model

9 December 201035

Social Network Analysis• Degree

• Centrality

– Closeness

– Graph

– Eigen Value– Eigen Value

• User Ranking

– Google Page Ranking

9 December 201036

Utterance evaluation

Social

Qualitative

• Degree

• Semantic similarity

• Predefined topics

• Overall discourse

Quantitative • NLP Pipe

• No of occurrences

)()())_log(1()( =mark(u) usocialuemphasisoccurencesnostemlengthwordsremaining

××

+×∑

)_,()_,()( keywordspredefineduSimdocumentwholeuSimuemphasis ×=

∏ +=

)(

))(log(1()(

equalitativandvequantitatiffactorssocialall

ufusocial

9 December 201037

Collaboration (1)

• Utterance graph

– Explicit links

– Implicit links

• Social cohesion

– Attenuation

– Trust

• Social cohesion

• Quantitative collaboration

explicit)(implicit/ links ofnumber total

trust(l)*n(l)attenuatio

=ioncollaborat vequantitatispeakersdifferent with

l links all∑

9 December 201038

Collaboration (2)

� Qualitative - Gain based collaboration =

ECHO

� Personal – individual knowledge building

� Links to previous utterances with same speaker

� Collaborative – collaborative knowledge � Collaborative – collaborative knowledge building

� Links to previous utterances with different speaker

∑speaker same have vandu

and utteranceearlier an is v v,andu between exists llink

trust(l))*n(l)attenuatio*v)(u,similarity*) gain(v)+((mark(v)=gain(u) personal

∑speakersdifferent have vandu and utteranceearlier an is v

v,andu between exists llink

trust(l))*n(l)attenuatio*v)(u,similarity*) gain(v)+((mark(v)=gain(u) ivecollaborat

9 December 201039

Collaboration (3)

9 December 201040

Services & Widgets� PolyCAFe is an online platform:

� Web services (Java and PHP-based)

� Web widgets using W3C Widgets1.0 standard

� The widgets can be integrated into any web platform that has a W3C widget container platform that has a W3C widget container (e.g. Wookie)

� There are services for maintenance tasks and analysis tasks (process discussion, search, etc.)

� The widgets have been deployed in Elgg (PLE)

9 December 201041

Presentation of Widgets

• Conversation Feedback (content + collaboration)

• Conversation Visualization (collaboration)

• Utterance Feedback (content + collaboration)

• Participant Feedback (content + collaboration)• Participant Feedback (content + collaboration)

• Search Conversation (content)

• Prepare chat/forum assignment

• Add conversations for assignment

9 December 201042

Conversation Feedback

9 December 201043

Conversation Visualization

9 December 201044

9 December 201045

9 December 201046

Search Conversation

9 December 201047

Validation and Verification of

PolyCAFePolyCAFe

9 December 201048

Validation Experiment 1.0

• Validation of PolyCAFe 1.0

– 9x students

– 5x tutors, 1x teacher

• Two chats with 4-5 students in a team

• All the students used PolyCAFe for the first time

• Only two tutors have previously used the system

• Students used the automatic feedback to get

insight about their discussion

• Tutors used PolyCAFe to offer manual feedback

9 December 201049

Validation Results – Tutors

� The tutors validated all the instruments with

suggestions to change part of them

� Quiz with 35 questions – all of them passed with an

average score between 3.50-5.00 / 5.00

� The system is relevant and useful for their � The system is relevant and useful for their

activity

� The time for providing final feedback to the

students is definitively reduced (between 30-

50%)

� The quality of the feedback is improved

9 December 201050

Validation Results – Tutors (2)

Category Average Perc. agree

Pedagogic effectiveness 4.11 83%

Efficiency 5.00 100%

Cognitive load 4.60 100%Cognitive load 4.60 100%

Usability 4.36 93%

Satisfaction 4.57 91%

Total 4.53 93%

1 – Strongly disagree; 2 – Disagree; 3 – Neutral; 4 – Agree; 5 – Strongly agree

9 December 201051

Validation Results – Students� The students have validated most of the

instruments� Quiz with 32 questions – 5 where not validated; all

others have scores over 3.66/5.00

� In the focus group, they reported that several misleads have been found using this widgetsmisleads have been found using this widgets

� These errors or misleads were reported to be only minor without influencing the overall feedback

� It has been suggested to try and fix them in order to gain the full trust of the users

9 December 201052

Validation Results – Students (2)

� For example, the students did not validate:

� “Overall, I believe that the support for my

learning PolyCAFe (Chat Analysis and

Feedback Service) provides is close enough

to the current support provided by humans. ”to the current support provided by humans. ”

� Average = 3.11

� Perc. agree = 33%

� However, we do not want to provide a

substitute for human evaluation!

9 December 201053

Validation Results – Students (3)

Category Average Perc. agree

Pedagogic effectiveness 3.94 77%

Efficiency 4.22 78%

Cognitive load 3.56 56%Cognitive load 3.56 56%

Usability 4.11 81%

Satisfaction 3.89 72%

Total 3.94 73%

1 – Strongly disagree; 2 – Disagree; 3 – Neutral; 4 – Agree; 5 – Strongly agree

9 December 201054

Validation Results - Widgets

Validation Statement Learners

Agreement Tutors

Agreement

PolyCAFe feedback is useful 100% 100%

PolyCAFe feedback is relevant 63% 80%

Conversation feedback is usefulConversation feedback is useful 78% 80%

Conversation visualisation isuseful

89% 100%

Utterance feedback is useful 83% 100%

Participant feedback is useful 78% 100%

Search conversation is useful 61% 100%

9 December 201055

Validation Experiment 2.0

� Validation of PolyCAFe 1.5

� 25x students in experimental group

� 10x students in control group

� 6x tutors in experimental group

� 2x 7 chats with 5 students in a team� 2x 7 chats with 5 students in a team

� Most of the students used PolyCAFe for the first time

� All the tutors have previously used the system

� Experiment is still under-way

� Preliminary results are very encouraging: very high correlation between ranking of participants by the system and by the participants themselves

9 December 201056

Verification� Accuracy of system ranking of participants

� Accuracy of grading utterances

� Accuracy of determining implicit links, collaboration areas, discussion threads

� Accuracy for determining speech acts� Accuracy for determining speech acts

� Accuracy for determining Model of Inquiry classes

9 December 201057

Example – Speech ActsSpeech act - label Precision Recall

Continuation 93% 92%

Statement 94% 93%

Greeting 100% 80%

Accept 92% 80%

Partial accept 71% 55%

Agreement 90% 51%

Understanding 96% 58%Understanding 96% 58%

Negative 97% 78%

Reject 73% 82%

Partial reject 35% 27%

Action directive 75% 90%

Info request 100% 71%

Thanks 100% 100%

Maybe 100% 69%

Conventional 66% 50%

Personal opinion 100% 36%

Sorry 66% 75%

9 December 201058

Conclusions and Future Work(PolyCAFe)

� The system is working well

� The system passed the validation with students and tutors

� The visualization widget proved to be the most usefuluseful

� The interface might be improved

9 December 201059

Transferability Issues• Domain

– The topic of the conversation should be easily

solved using discussions, no graphics or formulas

• Language

– Need for the components of the NLP pipe– Need for the components of the NLP pipe

– Corpus for training the LSA

– Maybe, a domain ontology

• Activity

– Collaborative activity

– Teams of 4-15 students (in the current design)

9 December 201060

Learning Support Based on Social

Networks Search and RecommendationNetworks Search and Recommendation

9 December 201061

Learning Scenario� Learner just uses social networking web sites

� He connects to friends and possible tutors

� Our application indexes the user’s friends and resources and their peers friends and resources

� Based on the data acquired we offer search and � Based on the data acquired we offer search and recommendation services

9 December 201062

Tag and Social Network-based Search

Community

=

Social networking application

Knowledge

=

Relevant content

Learner

=

Web 2.0 user

9 December 201063

Find Relevant Resources

Understand the learner’s request and lead him to a

resource that is both relevant and trusted as it is

recommended by a member of the community

9 December 201064

Find Relevant Peers

We search for the right person to offer feedback

and guidance

9 December 201065

Semantic Indexing of Learning Objects

• Crawled data

converted in semantic

formats •Vocabularies used: •Vocabularies used:

SIOC, Neuman’s tagging ontology,

SCOT, MOAT,

SKOS

•*using Amazon EC2

for crawling9 December 201066

Search and Recommendations

� Search

• Search algorithm based on PageRank

• Folkrank - a user is important if he annotates important

resources with relevant (well ranked) tags.

• Fokrank works on a folksonomy - hypergraph - F=(V,E)

∑∈

=)( )(

)()(

uBv vN

vRcuR

• Fokrank works on a folksonomy - hypergraph - F=(V,E)

where V=U ∪R ∪T and E⊂UxRxT

� Recommendation

• Uses clustering of resources, users and tags to

suggest relevant resources

9 December 201067

Architecture of the Application

9 December 201068

Using the Learning Services in a PLE

Search for

resourcesSearch for

users

username

Check the

learner’s

profile

Recommended

readings

Social networking platform

Relevant tags

and their

importance 9 December 201069

Conclusions and Future Work (Search and recommendations service)

� We use social data to provide feedback for learners

� We offer relevant resources to the user from his social network

� We help learners to find peers or tutors� We help learners to find peers or tutors

� The work was embedded in Moodle/WebCT and validated in Bucharest and Utrecht

� We’re centralizing the results

9 December 201070