Reference Model for Evaluating Intelligent Tutoring Systems
description
Transcript of Reference Model for Evaluating Intelligent Tutoring Systems
![Page 1: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/1.jpg)
1
Reference Model for Evaluating Intelligent Tutoring Systems
Esma Aimeur, Claude Frasson
Laboratoire HERONInformatique et recherche
opérationnelleUniversité de Montréal
![Page 2: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/2.jpg)
2Laboratoire HERON - TICE 2000
Introduction
• Goal of an ITS (Intelligent Tutoring System): produce the behaviour of an intelligent (competent) human tutor who can adapt his teaching to the learning rythm of the learner
• Ability to model and reason about domain knowledge, human thinking, learning processes, and teaching process
• building an ITS needs also to evaluate the system
• lack of evaluation methodology
![Page 3: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/3.jpg)
3Laboratoire HERON - TICE 2000
Architecture of an ITS
Curriculum
Interface
Didacticresources
Sessionmanager
Pedagogicalmodel
PlannerStudent model
Student
![Page 4: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/4.jpg)
4Laboratoire HERON - TICE 2000
Evaluation of ITS
• Evaluation can be formative or summative
• Summative: after the design process• Formative: during the design (define
and refine goals and methods)• Evaluation techniques used in
– ITS– Software engineering
![Page 5: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/5.jpg)
5Laboratoire HERON - TICE 2000
Evaluation of some ITS
• SHERLOCK : experimental evaluation• VCR Tutor : 4 versions compared• Formative qualitative evaluation
– iterative design– formative methods
• qualitative : characteristics of a situation• quantitative : finding causes and consequences
– case studies
![Page 6: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/6.jpg)
6Laboratoire HERON - TICE 2000
Lester’s evaluation
• On animated pedagogical agents (Design a Plant) --- botanical and physiology
• 100 middle school students• Important educational benefits :
improved problem solving• Better than less expressive animated
agents
![Page 7: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/7.jpg)
7Laboratoire HERON - TICE 2000
Littman- Soloway
• What is the educational impact on students ?
• What is the relationship between the architecture of an ITS and its behavior ?
• External evaluation of the student model (Proust)
• Internal evaluation : architecture, how ITS respond to input values
![Page 8: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/8.jpg)
8Laboratoire HERON - TICE 2000
Van Lehn
• Teaching metacognitive skills to implement and evaluate an ITS (SE-Coach guide self-explanation)
• Empirical evaluation• Fundamental questions
– How the design tools are used efficiently ?– How the learner improve his knowledge ?
![Page 9: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/9.jpg)
9Laboratoire HERON - TICE 2000
Software engineering techniques
• Product : Boehm’model – software doing what the user want it to do– use resources correctly– easy to learn– well designed, code– easily tested and maintained
![Page 10: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/10.jpg)
10Laboratoire HERON - TICE 2000
HCI techniques
• Evaluating design– cognitive walthrough (how easy a system
is to learn)– heuristic evaluation (visibility, consistency,
flexibility, helps for the user)– model-based evaluation (GOMS) predict
user performance
![Page 11: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/11.jpg)
11Laboratoire HERON - TICE 2000
Evaluationg Implementation
• Empirical and experimental methods• Observational methods (user
completes a set of tasks)• Query techniques
– interviews of the users about their experience
– questionnaires : questions fixed in advance
![Page 12: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/12.jpg)
12Laboratoire HERON - TICE 2000
Learner model
• Layers : – cognitive– affective– inferential
• Learning time : speed of knowledge acquisition
• Tracability : of learner’s actions
![Page 13: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/13.jpg)
13Laboratoire HERON - TICE 2000
Knowledge Level
Explication% of
expertise
Novice No prior knowledge of the subject at all, never introduced to the subject before
0%
Beginner
Familiar with the subject. Knows some of the rules but lacks in practice, expected to answer basic questions correctly.
10-30 %
Intermediate
Learner knows most of the rules and is expected to answer correctly half of the question, while trying to perform in the other half.
40-60 %
Expert
Completely knows rules. Have ability to answer most of the questions correctly. Mainly uses the system to make his knowledge perfect.
80-100 %
Learner model
• Levels of knowledge – Based on works of
Gagné (1985)– 7 levels of knowledge– 4 principal levels
• Novice • Beginner• Intermediate• Expert
![Page 14: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/14.jpg)
14Laboratoire HERON - TICE 2000
One-on-one
Co-learner
Learning Companion
Learning by Disturbing
Learning strategies
![Page 15: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/15.jpg)
15Laboratoire HERON - TICE 2000
Learning strategies
• Diversity• Adaptability (switching to different
strategies)• Modification• Memorization• Feedback• Reduce cognitive load
![Page 16: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/16.jpg)
16Laboratoire HERON - TICE 2000
Curriculum
• Generalisation• Consistency• Knowledge articulation• Reusability• Navigability• Maintenance
![Page 17: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/17.jpg)
17Laboratoire HERON - TICE 2000
Interface
• Intuitive : learner should understand easily the functions to apply
• Interactivity : allowing the learner to be active in his learning environment
• Matching : using words, phrase and concepts
![Page 18: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/18.jpg)
18Laboratoire HERON - TICE 2000
General
• Productivity rate : the most important factor. Time spent to produce one hour of ITS based course
• Learning outcomes : performance obtained by the learner after the session
• Ease of use of the tools
![Page 19: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/19.jpg)
19Laboratoire HERON - TICE 2000
Conclusion
• Evaluation process is complex but needs to be realistic
• Develop first , evaluate later• Evaluation criteria as a guideline for
estimation
![Page 20: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/20.jpg)
20Laboratoire HERON - TICE 2000
Merci de votre attention
![Page 21: Reference Model for Evaluating Intelligent Tutoring Systems](https://reader036.fdocuments.us/reader036/viewer/2022062423/56814cfe550346895dba238b/html5/thumbnails/21.jpg)
21Laboratoire HERON - TICE 2000
Stratégies d ’apprentissageStratégies d ’apprentissage
Learner
Co-Learner
Learner TeacherCompanion
LearnerTeacher
TeacherLearner
Tuteur Co-apprenant
Compagnon Apprentissage par explication