1
Evaluating the Effect of Predicting Oral Reading Miscues
Satanjeev Banerjee, Joseph Beck, Jack MostowProject LISTEN (www.cs.cmu.edu/~listen)
Carnegie Mellon University
Funding: NSF IERI
2
Why Predict Miscues?
• Reading Tutor helps children learn to read.• Speech recognizer listens for miscues (reading errors)
– E.g.: listen for “hat” if sentence to be read has word “hate”
• Accurate miscue prediction helps miscue detection.
3
Real Word Substitutions
• Miscues = substitutions, omissions, insertions
• Real word substitution = misread target word as another word– E.g. read “hat” instead of “hate”
• Most miscues are real word substitutions
• ICSLP-02: predicted real word substitutions
• Here: evaluate effect on substitution detection
4
How Evaluate Substitution Detection?
What child said I hat having milk every a
Correct text I hate having milk every day
What ASR heard I hate having mill every hate
substitution undetected false alarm substitution detected
substitution
1
2Substitution detection rate =
# substitutions detected
# substitutions child made=
1
4False alarm rate =
# false alarms
# words correctly read=
5
Evaluation Data
• Sentences read by 25 children aged 6 to 10
Correctly read Incorrectly read
Content tokens
5,981 335
Function tokens
3,166 147
6
Rote Method• Uses the University of Colorado miscue database.• For each target word
– Sort substitutions by # children who made them.
– Predict that the top n substitutions will reoccur, for this word.
Model typeSubstitution
detection rateFalse alarm
rate
No predicted substitutions
21.58 % 2.42 %
Top 1 22.82 % 2.88 %
Top 2 24.90 % 3.27 %
7
Extrapolative Method• Predict the probability that a word is a likely
substitution for another word– Pr ( substitution “hat” | target “hate”)
• Use machine learning to induce a classifier• Train using University of Colorado miscue database.
Some features (more in paper)
Candidate substitution
Target word
Spelling edit distance = 1 H.A.T.E H.A.T
Phonetic distance = 1 /HH EY T/ /HH AE T/
Rank in descending sorted frequency table
1364 1887
8
Extrapolative Method cont’d
Given a target word, predict substitution if
Pr ( substitution candidate | target word ) > threshold
Model typeSubstitution
detection rateFalse alarm
rate
No predicted substitutions
21.58 % 2.42 %
Pr >= 0.99 23.03 % 2.77 %
Pr >= 0.95 24.48 % 3.67 %
9
Combining Rote and Extrapolative
• Aim: Get n substitutions for a given word.
• Step 1: Use top n substitutions from rote.
• Step 2: If rote predicts k substitutions, k < n, – Then add top n – k substitutions from extrapolative.
• Intuition: rote is more accurate, so use when available. If not available, fall back on extrapolative.
10
Results from Combining Algorithms
Model typeSubstitution detection rate
False alarm rate
Top 1 25.73 % 3.77 %
Truncation 24.69 % 4.29 %
Top 2 31.54 % 4.62 %
Theoretical max
42.53 % 2.90 %
Truncation = The first 2 to n-2 phonemes of a word – models false starts. [/K AE/ and /K AE N/ for /K AE N D IY/; none for “hate”]
Theoretical max = use only those miscues the child actually made.
11
Conclusion
• Evaluated effect on substitution detection of – Two previously published algorithms– A combination of the two algorithms.
• Combined approach improved on current configuration (truncations) by– Reducing false alarms by 0.52% abs (12% rel)– Increasing miscue detection by 1.04% (4.2% rel)
• Take-home sound byte: Listening for specific reading mistakes can help detect them!
Top Related