Exploring the Potential of Speech Recognition to Support Problem Solving and Reflection

Manolis Mavrikis, Beate Grawemeyer, Alice Hansen, and Sergio Gutierrez-Santos

London Knowledge Lab

Wizards go to school in the elementary maths classroom

Aim

• Enabling natural interaction with a learning platform through multimodal interaction - speech.

• Wizard-of-Oz study to – Collect realistic data – Inform the design of the system – explore children’s inner speech– reflection as a learning mechanism

Voice interaction for learning• Educational research has shown benefits of verbalization of learning (1,2,3).

• More natural and efficient form of communication will have positive learning gains (4).

• Verbal communication in learning mathematics is important, as it is similar to learning a foreign language.

• Self-explanation can be beneficial for learning (5) as it can be seen as a tool to address student’s own misunderstandings.

• Speech can also provide cues for drawing inferences on students emotions and attitude toward the learning situation

Wizard-of-oz study

wizard

student

(1)

(6)(2)

(4)(3)

(5)

(5)

WOZ Setup

Ecological validity

Ecological validity

Wizard-of-oz study

Wizard-of-oz study

• Overall 60 students aged between 9 – 10 took part in a series of sessions with the iTalk2learn platform.

• 12/10 Students were wizarded for around 30 minutes.

• The students that were wizarded answered a questionnaire and took part in a focus group.

Fractions Lab – exploratory learning environment

Fractions Lab – exploratory learning environment

Wizard’s script for support

• Talk aloud• “Remember to talk aloud, and tell us what are you thinking”• “What is the task asking you to do?”• “Please think aloud, what are your thoughts or feelings?”

• Talk maths• “Can you explain that again using the terms denominator, numerator?”

• Affect • “It may be hard, but keep trying”• “If you find this easy, check your work and change the task”

• Problem solving • “You can’t add fractions with different denominators”

• Reflection• “What did you learn from this task?”• “What do you notice about the two fractions?”

Feedback types

Wizard of Oz results

• 170 messages were sent to 10 students

Affect Talk aloud Talk maths Problem solving

Reflection0

10

20

30

40

50

60

70

80

Wizard of Oz results

• Was the feedback immediately related to what the student said?

Affect Talk maths Problem solving Reflection0

10

20

30

40

50

60

70

80

NoYes

Wizard of Oz results

• Student reacted?

Affect Talk aloud Talk maths Problem solving

Reflection0

10

20

30

40

50

60

70

80

NoYes

Wizard of Oz results

• No significant correlation between the provision of feedback immediately after what the student said, and student’s reaction (r=.18, p>.05).

• However, there was a significant correlation if problem solving support was provided (r=.16, p<.05).

Questionnaire5

4

3

2

1

helpful repetitive understand like think aloud

Discussion• “Now you are talking”

– An ecological valid setting is important.

• “Sarcasm doesn’t become you”– Voice production is important.

• “Can the headsets look in my head?”– Students attribute higher abilities to the system.

• “Hmm – let me rephrase that “– Prompting students to use mathematic vocabulary might help to think

more carefully what they are saying.

• “Talking the talk”– Individual differences.

Conclusion

• The results indicate that there is potential in expecting young students to think-aloud while interacting with educational technology.

• Even rudimentary ASD has the potential to support basic reflection on the learning task and of domain terminology.

• Future work includes identifying how interruptive the messages are and carefully selecting timings on reflective prompts.

References1. Askeland, M.: Sound-based strategy training in multiplication. European Journal of Special Needs Education 27(2), 201-217 (2012)

2. Borasi, R., Siegel, M., Fonzi, J., Smith, C.: Using transactional reading strategies to support sense-making and discussion in mathematics classrooms: An exploratory study. Journal of Research in Mathematics Education 29, 275-305 (1998)

3. Mercer, N., Sams, C.: Teaching children how to use language to solve maths problems. Language and Education 20(6), 507-528 (2007)

4. Mayer, R.E., Moreno, R.: Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist 38(1), 43-52 (2003)

5. Chi, M.: Self-explaining expository texts: The dual processes of generating inferences and repairing mental models. In: Glaser, R., (ed.) Advances in Instructional Psychology, 161-238 (2000)