Effective Methods for Educational Research

The value of prototyping, observing, engaging users

Diana Laurillard

Methods for educational software designAn iterative design-test-analyse-redesign approachThe focus is achievement of learning outcomesAlign outcomes – activities – assessment• Discover requirements – design – design tests• Prototype• Observe• Test• Engage• Evaluate

Prototyping

• Sketch the design idea• Design user tasks– Use materials to represent choices and actions– Allow users to construct the choices and actions

• Data collection– 6 target users (min)– User constructions (materials)– Users’ commentary (audio, observation notes)– Video (usually unnecessary)

A paper prototype

Proposed screen design

Proposed actions

A paper prototype

User defined actions

A paper prototype

Observing

• Individual– Observation notes– ‘Stimulated recall’ (Bloom1953) – video or

playback to act as a memory aide – Exercise: ‘what were you thinking at this point?’– Audio + notes

• Pairs – listen to them talking, audio + notes• Recording of screen activities • Video

Data capture and analysis tools

To capture screen activities

To capture eye movements, logged interactions, and for eye-tracking data analysis

To coordinate and analyse the multiple streams of data

User engagement

• Workshops – with frequent discussions• Focus-group - question then facilitated

discussion• Survey – Closed questions + open comment• Construct elicitation interviews to discover

users’ own categories of description

Learning analytics

Measures that correlate with final grade• Total discussion messages posted (+ve)• Total number of online sessions (+ve)• Total time online (+ve)• Time spent on assignments (+ve)• Time spent on assessments (-ve)

(MacFadyen and Dawson, 2010)

LMS tracking those at risk

Ave grade = 72%Ave grade = 63%Ave grade = 57%Ave grade = 44%

(MacFadyen and Dawson, 2010)

Predicting those at risk

Week of course

Aver

age

hour

s on

line

per w

eek

Learning analytics on outcomesPedagogically-driven analytics• Define the ecology of experiences and

resources in which the software is embedded• Design appropriate assessment of outcomes • Use the software tasks to generate tests• Formulate the pre-post tests that will capture

the differences the software should make• Interpret outcomes for re-design

Working groups

• 1. What kinds of problems do you see with methods of prototyping, observing and engaging?

• 2. What might be the advantages of using such methods?

• 3. How might an educational software developer can draw on evidence-based research?