Beam Loss Monitor System – Preliminary Feedback November 11 th, 2010.
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback...
-
Upload
traian-rebedea -
Category
Education
-
view
630 -
download
0
description
Transcript of Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback...
Barcelona, 30th September 2010 5th European Conference on Technology Enhanced Learning, EC-TEL 2010
Overview and Preliminary Results of Using PolyCAFe for Collaboration
Analysis and Feedback Generation
Traian Rebedea, Mihai Dascalu, Stefan Trausan-Matu et al.
“Politehnica” University of Bucharest, Department of Computer Science and Engineering
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Overview
• Context: CSCL • Learning scenario: problems and solution• Our solution: PolyCAFe
– System architecture & Functionality– Words, voices & threads– Services & Widgets
• Validation Experiment• Transferability• Conclusions
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Context: CSCL
• Collaborative learning through dialogue• Most popular web technologies: chats and forums• Take into account the content of the discussion• But collaboration is also important
• No tools for analyzing these online discussions• The need for automatic assessment and feedback• Due to the high number of participants, parallel
discussion threads may appear!
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Learning Scenario
• Formal context– Undergraduate students in Computer Science– HCI course
• Students divided in small groups– Provided a topic for the discussion– First phase: document the topic, read materials– Second phase: participate in a chat conversation– Third phase (optional): receive feedback from tutors
• Similar scenario might arise naturally in an informal context (students or CoP)
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Problems
• Tutors find difficult the analysis of the chat conversations– Multiple participants– Multiple discussion threads– Difficult to remember who said what and when (what was
the context of the conversation)
• Very time consuming for tutors• The feedback provided by different tutors is very
heterogeneous• Students receive feedback very late
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Solution
• A software solution that:– Provides preliminary feedback and support to the
learner– Helps the tutors understand better and assess
quicker a conversation– The preliminary feedback is used by the tutor to
provide the final feedback to the learner– Offers improved search facilities to the users
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Proposed Solution - PolyCAFe
• Chat and Forum Analysis and Feedback System• PolyCAFe = Polyphony-based Collaboration Analysis
and Feedback generation• Provides feedback to learners, tutors and teachers
related to the interaction of students in online discussions
• Takes into account both:– Content of the conversation (related to a domain or topic)– Collaboration (related to the conversation, participation,
etc.)
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
System Architecture
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Technologies & Techniques
• Natural Language Processing– NLP pipe: spelling correction, phrase extraction,
stemming, POS tagger, NP-chuncker– Discourse analysis
• General elements: coreferences, argumentation• Conversation-related: speech acts
– Semantics• Linguistic ontologies (e.g. WordNet)• Tagged LSA (Latent Semantic Analysis)• Domain ontologies
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Words, Voices and Discussion Threads
• The topic of the discussion defines a list of important concepts (words/stems/…) => initial voices
• Other voices appear naturally during the conversation– Some may be related to the initial ones
• Voices can influence each other through explicit or implicit links between utterances
• One or more similar voices can define a discussion thread• Discussion threads = a set of utterances that are linked
implicitly or explicitly and correspond to a specific voice or more similar voices
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Functional Architecture
Social Network Analysis
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Utterance Evaluation
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Services & Widgets
• PolyCAFe is an online platform:– Web services– Web widgets using W3C Widgets1.0 standard
• The widgets can be integrated into any web platform that has a W3C widget container (e.g. Wookie)
• There are services for maintenance tasks and analysis tasks (process discussion, search, etc.)
• The widgets have been deployed in Elgg (PLE)
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Presentation of Widgets
• Conversation Feedback (content + collaboration)• Conversation Visualization (collaboration)• Utterance Feedback (content + collaboration)• Participant Feedback (content + collaboration)• Search Conversation (content)
• Prepare chat/forum assignment• Add conversations for assignment
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Conversation Feedback
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Conversation Visualization
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Search Conversation
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
More Examples?
• Interested to find more?• Click to watch tutorial (screencast)• http://augur.wu.ac.at/wp5/wp51/ltfll-student.
htm
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Validation Experiment
• Validation of PolyCAFe 1.0– 9x students– 5x tutors, 1x teacher
• Two chats with 4-5 students in a team• All the students used PolyCAFe for the first time• Only two tutors have previously used the system• Students used the automatic feedback to get insight
about their discussion• Tutors used PolyCAFe to offer manual feedback
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Validation Experiment (2)
• Brief information about the system• Watch screencast designed as a tutorial• Use PolyCAFe for 40-90 minutes + think-aloud• Each user wrote down observations while using the
system• Questionnaire with more than 30 Likert questions• Focus groups – 60-90 minutes
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Validation Experiment (3)
• Monitored by two investigators that could only answer questions for usability issues– In order not to influence the users
• Most of the students were interested about the functionality of the system– How does it work? – What technologies?
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Validation Results – Tutors
• The tutors validated all the instruments with suggestions to change part of them– Quiz with 35 questions – all of them passed with an
average score between 3.50-5.00 / 5.00
• The system is relevant and useful for their activity• The time for providing final feedback to the students
is definitively reduced (between 30-50%)• The quality of the feedback is improved
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Validation Results – Tutors (2)
Category Average Perc. agreePedagogic effectiveness 4.11 83%Efficiency 5.00 100%Cognitive load 4.60 100%Usability 4.36 93%Satisfaction 4.57 91% Total 4.53 93%
1 – Strongly disagree; 2 – Disagree; 3 – Neutral; 4 – Agree; 5 – Strongly agree
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Validation Results – Students
• The students have validated most of the instruments– Quiz with 32 questions – 5 where not validated; all others
have scores over 3.66/5.00
• In the focus group, they reported that several misleads have been found using this widgets
• These errors or misleads were reported to be only minor without influencing the overall feedback
• It has been suggested to try and fix them in order to gain the full trust of the users
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Validation Results – Students (2)
• For example, the students did not validate:• “Overall, I believe that the support for my learning
PolyCAFe (Chat Analysis and Feedback Service) provides is close enough to the current support provided by humans. ”– Average = 3.11– Perc. agree = 33%
• However, we do not want to provide a substitute for human evaluation!
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Validation Results – Students (3)
Category Average Perc. agreePedagogic effectiveness 3.94 77%Efficiency 4.22 78%Cognitive load 3.56 56%Usability 4.11 81%Satisfaction 3.89 72% Total 3.94 73%
1 – Strongly disagree; 2 – Disagree; 3 – Neutral; 4 – Agree; 5 – Strongly agree
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Validation Results - Widgets
Validation Statement Learners Agreement
Tutors Agreement
PolyCAFe feedback is useful 100% 100%
PolyCAFe feedback is relevant 63% 80%
Conversation feedback is useful 78% 80%
Conversation visualisation is useful 89% 100%
Utterance feedback is useful 83% 100%
Participant feedback is useful 78% 100%
Search conversation is useful 61% 100%
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Transferability Issues
• Domain– The topic of the conversation should be easily solved using
discussions, no graphics or formulas• Language
– Need for the components of the NLP pipe– Corpus for training the LSA– Maybe, a domain ontology
• Activity– Collaborative activity– Teams of 4-15 students (in the current design)
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Conclusions
• PolyCAFe is a solution under development that uses a mixture of techniques:– NLP & SNA & Polyphony
• Adapted them to web conversations analysis• PolyCAFe 1.0 was found useful by students and
tutors• Tutors seem to agree more with the relevance of
PolyCAFe• Students might need some more time to
accommodate with the software
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Conclusions (2)
• PolyCAFe 2.0 focuses on:– Improving the usability especially for students– Fixing some of the errors discovered in this
validation round
• PolyCAFe can be transferred to other domains and languages– Easier for English at this point
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Open Points
• Maybe tutors are biased due to the time reduction for providing final feedback– Everyone loves to have more time for something
else
• The next validation is done for HCI and Professional Behavior in Medicine
• Looking for people interested to use PolyCAFe!
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
Test it!
• http://ltfll-lin.code.ro/ltfll/wp5/• Follow link to test PolyCAFe version 1.5• Still working for version 2
– release November 2010
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
More Information
• Polyphony theory of discourse for chats with multiple participants: – Trausan-Matu & Rebedea, CICLING 2010
• A more detailed technological and functional description of PolyCAFe– Dascalu, Rebedea & Trausan-Matu, AIMSA 2010
• Screencasts (video tutorials )for LTfLL services:– http://augur.wu.ac.at/screencasts/v1/
Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation
EC-TEL 2010, Barcelona30th September 2010
THANK YOU!
Questions &
Feedback