Evaluation of Tutoring Systems Kurt VanLehn PSCL Summer School 2006.
Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.
-
Upload
candace-banks -
Category
Documents
-
view
213 -
download
1
Transcript of Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.
![Page 1: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/1.jpg)
Sims for driving(50 mins)
CPI 494, April 29, 2009Kurt VanLehn
![Page 2: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/2.jpg)
Some examples
• Transom – driving a Remotely Operator underwater Vehicle (ROV).– Roberts, Pioch & Ferguson (2000) Verbal coaching during a
real-time task, IJAIEd 11, 377-388
![Page 3: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/3.jpg)
ROV
• Tethered, which affects motion• Camera, lights & sonar – limited visibility• Underwater currents
![Page 4: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/4.jpg)
Typical task
![Page 5: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/5.jpg)
Transom controlsJoystick controls thrusters
Camera imageSonarMapCompass headingDepthTether length
![Page 6: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/6.jpg)
Training phases
• Demonstration– Expert drives a mission & talk aloud
• Coached practice– During the mission: speech only (eyes & hands occupied)
• Student can ask for help, e.g., “bird’s eye view”• Unsolicited help e.g., “drifting too far north”
– After a mission• Replay• Measures• Path vs. expert’s path
![Page 7: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/7.jpg)
Students’ most popular help request: where am I?
![Page 8: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/8.jpg)
Three other help requests
• What should I be doing?– “Head directly toward the target, keeping it
constantly in view.”
• How do I do it?– “Follow your progress using the navigation plot.
Use side-stick to compensate for currents.”
• Tell me more.– Next hint along the hint sequence
![Page 9: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/9.jpg)
Pedagogical technology
• Detecting learning opportunities– Horizontal deviation from prescribed path– Depth deviation – Bearing deviation– Hover deviation (when hovering only)– Progress along path– Smoothness of thrust commands– Collisions
• Detectors drive coaching during the mission• Detectors drive after-action review
![Page 10: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/10.jpg)
Initial, simple pedagogy for unsolicited help
• If ROV detector fires, then alert the student• Failures
– Too disconnected– Need dialogue/hint sequence
• Check depth apply upward thrust
– Too repetitious– Too rapid
![Page 11: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/11.jpg)
Solution: Coaches replace “detectalert” rules
![Page 12: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/12.jpg)
Different coaches in different phases
• Phases– Approach phase– Orbit phase– Docking phase
PriorityPauseSensor
![Page 13: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/13.jpg)
Other pauses
• Grace period – how long sensor is out of bounds before coaching initiated.
• Refractory period per coach (7 second)• Pause between utterances regardless of
source
![Page 14: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/14.jpg)
During a phase
• All coaches are active, checking their sensor for learning opportunities
• Higher priority coach can interrupt a lower priority one that has started to given advice.– “New Problem!” signals the interruption
![Page 15: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/15.jpg)
Additional monitoring & feedback that was added later
• Overcorrection “Too far!”• Reverse correction “Wrong way!”
![Page 16: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/16.jpg)
Task loop
• Selection of next mission based on mastery of skills
![Page 17: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/17.jpg)
Step loop
• Monitoring continuous “steps”• Detectors find learning opportunities; start
coach• Coach gives immediate feedback and hints• Assessment of student’s skill updated
continously
![Page 18: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/18.jpg)
Evalution
• Transom• Simulated ROV only – fixed mission sequence• ROV only (not simulated) – fixed mission seq.• No training• Experts on the post-test
![Page 19: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/19.jpg)
Pre and post-tests
• Maneuvering task used in pre, training & post• Transfer task: pipe inspection• Sensor readings used to evaluation RMS
deviation
![Page 20: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/20.jpg)
Results for transfer task
Tutor should have produced lower
errors than untutored
![Page 21: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/21.jpg)
Aptitude treatement interactionPoor drivers
stayed poor with tutoring
Poor drivers got better without
tutoring
![Page 22: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/22.jpg)
Why did tutoring fail?
• Many tutees did not get to most challenging training tasks (high current; low visibility)
• Timing of coaching may have rewarded bad behavior
• Low ability students may have become dependent on the coaches
![Page 23: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/23.jpg)
TRIO: Trainer for Radar Intercept Operators
• Ritter & Feurzeg (1988) Teaching real-time tactical thinking. In Psotka, Massey & Mutter (eds) Intelligent tutoring Systems: Lessons Learned. LEA, 1988.
• RIO sits in back seat of F-14• RIO flys plane until bogey is in view
![Page 24: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/24.jpg)
Task
• RIO tries to fly “the gouge” • Planes approach at high speed• RIO fires a long range shot at nose of bogey• Then a medium range shot at side of bogey• Then does a u-turn behind the bogey,
approaches it from the rear, fires a short-range shot at tailpipes of bogey
![Page 25: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/25.jpg)
Geometry
![Page 26: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/26.jpg)
Student’s view of the world
Bogey position, direction & airspeed relative to own
aircraft
Height, speed & heading
Own aircraft
![Page 27: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/27.jpg)
Student’s action
• Verbal commands to (simulated) pilot• “Come starboard hard as possible to a
heading of two four zero degrees”
![Page 28: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/28.jpg)
Demonstration
• Expert gives commands & explains why
![Page 29: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/29.jpg)
Coaching the student
• Different phases have different goals– During initial approach, get own speed to be 0.1
mach greater than bogey’s speed– If not achieving goal, detector fires & says “Get
co-speed plus”
• Mandatory pause between coaching utterances
• No hint sequences, just one warning.
![Page 30: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/30.jpg)
After action review
• Replay• Ground track• Critics
![Page 31: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/31.jpg)
Tutors for vehicle sims
• Three modes– Demonstration– Coached practice– Uncoached practice
• During a mission, there are phases• Each phase has different detectors for learning
opportunities (i.e., errors) • Detectors watch for parameters to go out of bounds,
for controls to be set wrong or ignored…
![Page 32: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/32.jpg)
Pedagogy
• During the mission– Keep it short & verbal– Need pauses– Need hint sequences instead of repetition– Need priorities
• During the after action review– Ground track / truth– Prioritize learning opportunities– Show difference & exaplain
![Page 33: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/33.jpg)
Step-based tutoring?
While not expert do• Select task/mission• Drive the mission – first step loop?• After action review – second step loop?• Update student model/profile
![Page 34: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/34.jpg)
Step loop while driving the mission
• Continuous action, not discrete• Learning opportunities = errors = detectors
fire• Immediate feedback, sometimes • Hint sequences, sometimes• Watch out for cognitive overload• Watch out for becoming overly dependent on
detectors
![Page 35: Sims for driving (50 mins) CPI 494, April 29, 2009 Kurt VanLehn.](https://reader036.fdocuments.us/reader036/viewer/2022070411/56649f4f5503460f94c71451/html5/thumbnails/35.jpg)
Step loop during after-action review
• Breaks up the mission into phases• Breaks up a phase into errors & ok• Gives feedback on errors (learning
opportunities)