Kate’s Ongoing Work on Uncertainty Adaptation in ITSPOKE.

Kate’s Ongoing Work on

Uncertainty Adaptation in ITSPOKE

Outline

PSLC 2006 - 2007 Project

ALT 2007 - 2009 Project

Differences from PSLC Project

NAACL-HLT 2007 Study

PSLC Project: Background Does Treating Student Uncertainty as a Learning Impasse

Improve Learning in Spoken Dialogue Tutoring?

Learning Impasse (VanLehn, Siler & Murray, 2003):

“occurs when a student realizes that he or she lacks a complete understanding of a specific piece of knowledge [...] when a student gets stuck, detects an error, or does an action correctly but expresses uncertainty about it”

uncertain answers and wrong answers

PSLC Project: Background Does Treating Student Uncertainty as a Learning Impasse


Uncertainty relates to learning (Craig et al. 2004):

positively correlates with learning in AutoTutor

Hypothesis: uncertainty can be accompanied by deliberation/inquiry aimed at increasing understanding

PSLC Project: Hypothesis Does Treating Student Uncertainty as a Learning Impasse


uncertain: opportunity to learn something incorrect: opportunity to learn something

Most tutoring systems respond only to correctness

Our hypothesis: treating uncertainty as a learning impasse will increase learning

PSLC Project: Experimental Condition Keep ITSPOKE response to incorrect answers:

Simple questions: give answer and move on

Complex questions: remediation dialogue with more questions to increase understanding, then move on

ADD ITSPOKE response to uncertain answers:

Respond as if answer were incorrect

Keep ITSPOKE response to other correct answers:

move on to next question

PSLC Project: Control Condition #1

Keep ITSPOKE response to incorrect answers

Keep ITSPOKE response to correct answers

NO ITSPOKE response to uncertainty

PSLC Project: Control Condition #2 Keep ITSPOKE response to incorrect answers

Keep ITSPOKE response to 85% correct answers

CHANGE ITSPOKE response to a random 15% of correct answers

Respond as if answer were incorrect

15% of student answers are uncertain (HC-wade)

control for extra tutoring in experimental condition; should be less effective than targeting uncertainty

PSLC Project: Experimental Procedure Read background material (reduced) Take pretest (reduced) Work 1st problem with ITSPOKE (#58) Take posttest (reduced)

Tests modified to be “fill in blank” (no chance correct) May not see significant learning gains after 1 problem

Work isomorphic problem with ITSPOKE (#59) Experimental condition may have more correct answers

20 subjects/condition (low pretest only); 2 hours

PSLC Project: Isomorphic Problems Problem #58: Suppose a man is in a free-falling elevator and

is holding his keys motionless right in front of his face. He then lets go. What will be the position of the keys relative to the man's face as time passes? Explain.

Problem #59: A professor is sitting in his armchair holding his spectacles motionless in front of his face, in order to inspect how well he just cleaned them. All of a sudden, an earthquake causes his office floor to collapse. The professor is startled when he finds that both he and his armchair are in free-fall, and he drops his spectacles. What will be the position of the spectacles relative to the professor's face as time passes? Explain.

PSLC Project: Implementation WOZ version of ITSPOKE (“ideal” system conditions)

Speech input/output No essay (everyone gets walk-through)

Human Wizard Perform speech recognition and NLU (select anticipated

response components) Label uncertainty for each condition (checkbox; data)

Remaining Issues: students hit stop/start buttons when speak? random 15% of correct turns (non-uncertain?) need piloters so we can check logs etc ;)

PSLC Project

DEMO

ALT Project: Differences from PSLC

Adapting to Student Uncertainty over and above Correctness in a Spoken Tutoring Dialogue System

Experiment 1: 4 conditions, WOZ, all 5 problems Adaptation 1: treat uncertainty as learning impasse

(baseline/obvious/least-change strategy) Adaptation 2: new system responses to uncertain +

correct and uncertain + incorrect student answers, based (in part) on analysis of human tutor responses

Hypothesis: significantly improved learning in adaptive conditions as compared to controls (same 2 as PSLC)

ALT Project: Differences from PSLC

Adapting to Student Uncertainty over and above Correctness in a Spoken Tutoring Dialogue System

Experiment 2: 3 conditions, ITSPOKE, all 5 problems Adaptation: “best-performing” adaptation from Exp1

via learning, dialogue efficiency, survey, etc automatic uncertainty detection automatic speech recognition/NLU

Hypothesis: significantly improved learning in adaptive condition as compared to controls (same 2), but less improvement as compared to WOZ

ALT Project: Adaptation 2 Adaptation 2: new system responses to uncertain + correct and

uncertain + incorrect based (in part) on human tutor responses Pon-Barry et al., 2006: implemented human tutor responses,

but didn't find increased learning in experimental condition uncertain + correct: paraphrase answer

corpus#1: 4/5 and 0/34 other corrects corpus#2: 10% and 1% other corrects

uncertain + incorrect: refer to past dialogue corpus#1: 3/10 (point out misconception, follow

incorrect reasoning, state answer) and 0/7 other corrects

corpus#2: 30% and 5% other incorrects Human tutors use multiple strategies depending on context

NAACL07: Towards Adaptation 2 Exploring Context-Dependent Affect Adaptations in Spoken

Dialogue Systems (with Mihai, Diane, Joel) Proposal: develop multiple adaptations for affective state,

whose use depends on prior context in which state occurs Motivations:

Human tutors seem to do this (Pon-Barry et al., 2006) Prior work largely context-independent and no improved

learning (e.g., Aist et al., 2003, Pon-Barry et al., 2006) Call Center: apologize after user frustration, except if

multiple frustrated turns in row give to human operator

NAACL07: Towards Adaptation 2 Exploring Context-Dependent Affect Adaptations in Spoken

Dialogue Systems (with Mihai, Diane, Joel) Questions: in what contexts to adapt, and what strategy to

use in each context? (not clear for uncertainty in tutoring) Our Empirical Approach:

Apply Chi2 test is system corpora to find contexts significantly associated with increased/decreased affect

Formulate hypotheses explaining association Base context-dependent adaptations on hypotheses Refine adaptations via analysis of human tutor responses

NAACL07: Towards Adaptation 2 Apply approach in ITSPOKE Targeted affect: uncertain+correct, uncertain+incorrect Contexts are question properties: Question Act, Discourse

Structure Depth/Transition, Goal Repetitions “Tutor Question – Student Answer”, but 275 questions

Results: multiple contexts significantly associated with increased

or decreased uncertainty states hypotheses explain association and suggest adaptations

FW: Refine via analysis of human tutor response in contexts

NAACL07: Annotated ExampleSYS4: Good. Does the packet have an acceleration? If yes please specify its

direction. [SAQ, Advance, D1, GR0]

STUDENT4: horizontally forward yes [other+incorrect]

SYS5: Well... To find the packet's acceleration we will use Newton's

second law. What is the definition of this law? [LAQ, Push, D2, GR0]

STUDENT5: an object in motion tends to stay in motion until it's act

by an outside force [uncertain+incorrect]

SYS6: Well... [...] What can we determine about the acceleration of the

packet given that we know the net force acting on it is not zero and

points straight down? [DAQ, Advance, D2, GR0]

STUDENT6: I don't know [uncertain+incorrect]

SYS7: Well... [...] Does the packet have an acceleration? If yes please specify its direction. [SAQ, PopUp, D1, GR1]

Question Act Dependencies (p<.05)

LAQ/DAQ are hard questions: definition/deep reasoning Adaptation: reformulate as ``fill in blank” SAQ to work

through uncertainty by (also) answering simpler version correctly

Dependency Obs. Exp. Chi^2QACT ~ SSTATE 203.38LAQ ~ uncC + 72 22 133.98RPT ~ uncC - 7 14 4.15SAQ ~ uncC - 285 328 41.95DAQ ~ uncI + 37 27 4.57LAQ ~ uncI + 43 27 11.17SAQ ~ uncI - 377 408 17.1

Question Act Dependencies (p<.05)

SAQ are easier “fill in blank” questions: uncertain students may be having difficulty with path to solution

Adaptation: remind how SAQ relates to overall solution (and/or monitor frequency of uncertainty after SAQs)

Dependency Obs. Exp. Chi^2QACT ~ SSTATE 203.38LAQ ~ uncC + 72 22 133.98RPT ~ uncC - 7 14 4.15SAQ ~ uncC - 285 328 41.95DAQ ~ uncI + 37 27 4.57LAQ ~ uncI + 43 27 11.17SAQ ~ uncI - 377 408 17.1

Depth Dependencies (p<.05)

Deeper depths correspond to deeper knowledge gaps about topics in solution (+otherI, -otherC): uncertainty may concern perceived lack of cohesion between sub-topic and larger topic/solution

Adaptation: show how sub-topic fits in to overall solution

Dependency Obs. Exp. Chi^2D# ~ SSTATE 53.85D1 ~ uncC + 250 228 5.46D2 ~ uncC - 78 101 7.8D1 ~ uncI - 230 283 27.55D2 ~ uncI + 156 125 11.26D3+ ~ uncI + 93 70 9.76

Depth Dependencies (p<.05)

D1 on direct path to solution (+otherC, -otherI): uncertainty may relate to difficulty understanding necessary steps on path

Adaptation: reminder of steps on path, and/or monitor frequency of uncertainty at D1: occasional may be constructive (VanLehn et al., 2003) but frequent may be destructive and require intervention

Dependency Obs. Exp. Chi^2D# ~ SSTATE 53.85D1 ~ uncC + 250 228 5.46D2 ~ uncC - 78 101 7.8D1 ~ uncI - 230 283 27.55D2 ~ uncI + 156 125 11.26D3+ ~ uncI + 93 70 9.76

Transition Dependencies (p<.05)

Pushes correspond to deeper knowledge gaps (+otherI): uncertainty may concern perceived lack of cohesion between sub-topic and larger topic/solution (like D<1)

Adaptation: show how sub-topic fits in to overall solution

Dependency Obs. Exp. Chi^2TRAN ~ SSTATE 190.97PopUp ~ uncC - 23 36 5.89Advance ~ uncI - 223 254 9.06NewTopLvl ~ uncI - 49 67 6.47PopUp ~ uncI - 32 45 4.68PopUpAdv ~ uncI + 32 17 16.22Push ~ uncI + 100 70 16.37SameGoal ~ uncI + 43 25 14.24


After PopUpAdv, uncertainty may relate to losing track of original question (not +otherI); after PopUp, to goal of sub-topic question(s)

Adaptation: remind how PopUpAdv relates to original question; clarify PopUp's subtopic goal



Uncertainty may relate to current topic/depth(-otherI) Adaptation: can be based on other contexts simultaneously

monitored (e.g., Depth, QACT)



SameGoal associated with increased incorrect overall; RPT with decreased correct: if student repeats answer, uncertainty may relate to original answer

(Rotaru and Litman, 2006): RPTs should be removed


Goal Repetition Dependencies (p<.05)

Uncertainty may be increased simply because students haven't seen question before (not +otherC); some may be constructive

Adaptation: can be based on continued uncertainty at GR0 (monitor frequency like after SAQs, D0 )

Dependency Obs. Exp. Chi^2GR# ~ SSTATE 20.59GR0 ~ uncC + 301 281 5.46GR2 ~ uncC - 7 14 3.93

Goal Repetition Dependencies (p<.05)

Most students may remember answer from last time it was asked (no uncertainty) (not +otherC)

Adaptation: remind students about last time question asked (akin to Pon-Barry et al., 2006)

Dependency Obs. Exp. Chi^2GR# ~ SSTATE 20.59GR0 ~ uncC + 301 281 5.46GR2 ~ uncC - 7 14 3.93

Conclusions & Future Work Approach shows dependencies between uncertainty states

and contexts, suggests hypotheses leading to adaptations Will refine adaptations by analysing human tutor

responses to uncertainty states in these contexts Will study other automatically-monitorable contexts:

Topic Repetition: track similar + identical questions about topics, within and across 5 SYS problems (e.g. gravity, freefall); topic-dependent adaptations can be derived if uncertainty relates to specific topics

Will develop Combined/Precedence rules when multiple contexts trigger context-dependent adaptations

Kate’s Ongoing Work on Uncertainty Adaptation in ITSPOKE.

Documents

Transcript of Kate’s Ongoing Work on Uncertainty Adaptation in ITSPOKE.