Post on 19-Jun-2015
description
An Initial Evaluation ofMetacognitive Scaffolding for
Experiential Training Simulators
Marcel Berthold, Adam Moore, Christina Steiner, Conor Gaffney, Declan Dagger, Dietrich Albert, Fionn Kelly,
Gary Donohoe, Gordon Power, Owen Conlan
• Who?• ImREAL Project• Evaluating Self-Regulated Learning and Metacognitive Scaffolding in Experiential
Simulator
• How?• Cross cohort comparison
• Baseline usage of sim vs scaffolded usage• Initial – no cross correlation / matching• Third year medical students
• What (did we find out)?• Performance• SRL behaviour• Scaffolding encounters & effects• Analysis of reflection text New! (Not in paper)
Overview
• create effective virtual reality training simulation tools that adapt to trainees’ past experiences or preconceptions
• closing the gap between the ‘real-world’ and the ‘virtual-world’
• Services should respond to users’ behaviour and adapt accordingly to the user model based on a pedagogical model
• Create services that can be connected to different simulators
Rationale of ImREAL
What do we want to know?
1. Is self-regulated learning supported?
2. Does the simulator augmentation through the service lead to better learning performance?
3. Does the simulator augmentation through the service increase motivation?
4. Is the service well integrated in the simulation and learning experience?
• SRL is composed of three cyclic learning phases: Forethought, Learning, Reflection (Zimmermann, 2002)
• Good SR learners use appropriate learning strategies and techniques
• Good SR learners achieve
better learning results and
are more motivated to learn
(Zimmerman, 2002; Veenmann, 2011)
SRL@ET workshop Kato Galatas, 15-06-2012
Self-Regulated Learning (SRL)
• Integrated SRL cycles + Peer experience
(Hetzner et al., 2011)
Pedagogical framework in ImREAL
Simulator
• Dialogic• Training for diagnostic interviews for psychiatric patients
Reflection Elicitation / Scaffolding
• Simulator has pre-existing reflection tool
• Can be triggered by learner• Prompts to reflect also added by
instructional designer
• In practice mode, cohort 2 students had ImREAL Metacognitive Scaffolding Service (MSS)
Phase 1 Trial
Scaffolding – Thinking Prompts
• Approximately 140 (cohort 1 = 131 / cohort 2 = 143) medical students, on average 21 years old (40% male vs. 60% female, 80% Irish). They performed the simulation as part of their third year medical training at Trinity College, Dublin (cohort 1 in 2011, cohort 2 in 2012).
• Cohort 2 additionally has scaffolding prompts in practice mode only
• Experience• No experience with ETU simulator• Experienced with interviews (97 %)• limited experience with interviewing psychiatric patients (15 %)
Cohorts
User
Reflection postings
Questionnaire Data
Log-Data
Assessment
Have you set a learning goal?
Methods
• Project evaluation question
o Can SRL be enhanced through
Metacognitive Scaffolding Services (MSS)?
• Formative evaluation approach
• Research foci of ImREALMSS
Integrated ImREAL services
• Investigate: Impact on SRL
reports, behaviour, qualitative
feedback
http://www.empowertheuser.ie
Methods: Evaluation approach in ImREAL (First User trail)
• No change in SRL compared to baseline• Highest usage of elaboration strategies
Memorizing strategies
Elaboration strategies
Organisation strategies
Planning strategies
Self-monitor-ing strategies
Time man-agement strategies
Achievement motivation strategies
Internal attri-bution
Effort strate-gies
First user trial
57.6455026455026
70.15873015873
62.6772486772486
59.1375661375661
60.005291005291
61.2169312169312
62.4047619047619
59.1111111111111
55.7777777777778
Baseline Evalua-tion
54.0687134502924
69.7953216374268
62.2368421052631
58.2353801169591
60.9078947368421
59.9269005847953
60.4385964912281
56.8728070175438
55.3004385964912
5
15
25
35
45
55
65
75
85
95
SRL strategies profil
SRL strategy scales
%
t20=3.34, p=.003
Results: SRL
• Scores are above centre line of the rating scale• Mentioned that prompts were not always presented at appropriate time
1
2
3
4
5
3.42 3.333.09
3.392.97 3.12 3.27 3.38 3.45 3.42 3.26
Learning Experience MSS
Items
Ra
tin
gs
Results: Learning Experience
• Students were motivated• To perform simulation• Learn about interviews• To apply skills learnt in simulation in real world
• State motivation after simulation higher in 1st user trial compared to baseline
Baseline Evaluation First User Trial0
1
2
3
4
2.48
3.35
Motivation
Trial
Ra
tin
gs
t118.47=-8.64, p<.001
Results: Motivation
• Comparison of expected and empirical MSS prompt distribution• More scaffolds in Information Management phase• Less scaffolds on reflection
Planning Informantion Management
Monitoring Debugging Reflection
Expected distri-bution
400.2 571.714285714286
400.2 301 343.028571428571
Empirical distri-bution
469 752 425 301 54
50
150
250
350
450
550
650
750
Expected vs. Empirical Distribution of MCSs
Phases
Fre
qu
enci
es
χ2(4,0.95)= 314.55, p<.001
Results: Scaffolding
• Positional – time, date, simulator, etc.• “Rapport and Initial inquiry have been completed. Must now concentrate on moving
the interview forward”
• Technical – material addressing the system, questions or underlying model • Great concept. Has a lot of potential, will be much better when more points to consider
and points of information are added to the overview that accurately reflect performance.
• Notes – observations of the patient • “Px. unable to quantify length of period she has been like this for.
• Reflections – reflective text • it is quite important to deal with the patient with empathy to make them comfortable
and also to try and illicit the cause of the bout of depression, in this case. I felt I did not do too bad as far as expressing empathy is concerned, allowing the patient to open up and try and formulate a management plan. More experience in history taking will be the key
Methods: Coding reflective text
• Corpus 1 (from cohort 1): had 358 unique text entries from 81 unique UserIDs giving an average of 4.41 entries per unique UserID• Engaged percentage = 57.9%
• Corpus 2 (from cohort 2): had 107 unique text entries into the note-taking tool from 50 unique UserIDs giving an average of 2.14 entries and 3418 views of the MST, resulting in 28 entries, 19 with text from 8 unique UserIDs• Engaged percentage = 35.7%
Results: Reflective Text
• percentage breakdown of each type of content within the note-taking tool across the two cohorts:
Position Technical Notes Reflection
Cohort 1 15 33 25 69
Cohort 2 17 57 16 66
Results: Reflective Text
• SRL• No correlation of log-data and SRL reports were observed – needs to be investigated
why? • No changes in SRL – long-term process
• MSS• Learners seem to need more assistance in effectively processing information by hints to
use more organizational, elaborative, summarizing or selective learning strategies• Are they more confident in the reflection phase and so pass the offer of scaffolds
• Time to engage• Less availability of simulators in cohort 2• Less lab time & shorter overall period
• Overall engagement and motivation increased• Higher percentage of time being scaffolded
Discussion
• Provide MS at appropriate times
• Keep learners longer in simulation
• Provide additional services to promote SRL
Lessons Learnt
What did we find out?
1. Is self-regulated learning supported?• Yes • Is it improved . . . . Maybe
2. Does the simulator augmentation through the service lead to better learning performance?
• No! But . . . . Yes!
3. Does the simulator augmentation through the service increase motivation?
• Yes
4. Is the service well integrated in the simulation and learning experience? • Yes
• Longitudinal study• SRL changes slow• Integration of changes into behaviour
• Fuller analysis of competency• Metacognitive / SRL as well as that trained by the simulator• Surfacing of reflection / competency – social?
• Affect support• Encouragement / engagement / motivation• Please take our survey!!!
Further Work
• The research leading to these results has received funding from the European Community's Seventh Framework Program (FP7/2007-2013) under grant agreement no 257831 (ImREAL project) and could not be realized without the close collaboration between all ImREAL partners.
Acknowledgements
http://bit.ly/smILEYCase sensitive!!!
Adam.moore@tcd.ie
@adam__moore
@ImREAL_project / www.imreal-project.eu
Thank-you!