ICCHP 2006 1

Adaptive Systems Research GroupUniversity of Hertfordshire

Megan DavisKerstin Dautenhahn

Chrystopher NehanivStuart Powell

TouchStory: Towards an TouchStory: Towards an Interactive Learning Environment for Interactive Learning Environment for

Helping Children with Autism to Helping Children with Autism to Understand NarrativeUnderstand Narrative

ICCHP 2006 2

Project AimsProject Aims

Children with autism have a deficit in narrative ability which adversely effects their social comprehension.

Our long term goal is to develop interactive systems which improve social comprehension in individual children with autism by enhancing their narrative abilities.

This study is concerned with the design of an interactive picture-narrative completion game for children with autism.

ICCHP 2006 3

AutismChildren with autism have

impaired social interaction impaired communication (verbal and non-verbal) impaired imagination

They form a diverse group but share a difficulty in making sense of the world, particularly the social world.

Children with autism generally like repetitive games may be very routine oriented may have particular sensory sensitivities may focus on seemingly irrelevant details (local cohesion)

ICCHP 2006 4

SquirrelRubbish bin

Ref: Story told by Rita Jordan

“Oh, look…”

ICCHP 2006 5

SquirrelRubbish bin

“….….someone has thrown away a perfectly good

squirrel.”

Ref: Story told by Rita Jordan

ICCHP 2006 6

Narrative and Social ComprehensionNarrative and Social Comprehension

► Narrative is fundamental to the perception, Narrative is fundamental to the perception, creation and communication of meaning in creation and communication of meaning in social interactionsocial interaction

► By fitting events into a narrative pattern we By fitting events into a narrative pattern we construct and inhabit a meaningful social construct and inhabit a meaningful social worldworld

► Narrative comprehension involves building an Narrative comprehension involves building an internal representationinternal representation Gaps in the narrative exposition must be Gaps in the narrative exposition must be

filled by experiential or semantic knowledgefilled by experiential or semantic knowledge

ICCHP 2006 7

The longitudinal studyThe longitudinal study►Aim: To investigate primitive aspects of Aim: To investigate primitive aspects of

narrative as a pre-cursor to narrative narrative as a pre-cursor to narrative comprehensioncomprehension Can an adaptive interactive system be Can an adaptive interactive system be

used to identify aspects of narrative which used to identify aspects of narrative which individual children find difficult? individual children find difficult?

Can an adaptive interactive system be Can an adaptive interactive system be used to enhance children’s understanding used to enhance children’s understanding of narrative?of narrative?

ICCHP 2006 8

Why use computers?Why use computers?

► Most children with autism enjoy using computersMost children with autism enjoy using computers highly predictable and controllablehighly predictable and controllable naturally monotropicnaturally monotropic may be highly visualmay be highly visual

► Easy logging, record keeping; timely analysisEasy logging, record keeping; timely analysis► Computers don’t get bored or irritableComputers don’t get bored or irritable► Quicker turn round, children are exposed to more Quicker turn round, children are exposed to more

good examplesgood examples

ICCHP 2006 9

Challenges of designing Challenges of designing software systems for children software systems for children

with autismwith autism► Usual methods (interviews, focus groups, Usual methods (interviews, focus groups,

collaborative design, etc.) not possiblecollaborative design, etc.) not possible children may have limited languagechildren may have limited language they typically find what? why? questions difficult and so they typically find what? why? questions difficult and so

answers are not helpfulanswers are not helpful they are not socially oriented -- collaborative or group they are not socially oriented -- collaborative or group

based work not appropriatebased work not appropriate► Local cohesion and the variability of autism both Local cohesion and the variability of autism both

mean it is difficult to predict what a child will find mean it is difficult to predict what a child will find difficultdifficult

► Repetitive behaviour issuesRepetitive behaviour issues

ICCHP 2006 10

Requirements Requirements elicitationelicitation

► explore the explore the children’s reaction children’s reaction using laminated using laminated cards and a replica of cards and a replica of the game using a the game using a touchscreentouchscreen

► 18 children (14 ASD), 18 children (14 ASD), ages 4-9 years, from ages 4-9 years, from two schoolstwo schools

► 5 stories from 5 stories from published picture published picture story booksstory books

Focussed: no extraneous sound or animation, rewards re-enforce the task

Designed to minimise meaningless repetition (autonomy/control balance)

Expect a desire for repetition so no need to design to keep attention

Builds on previous autism and technology research

ICCHP 2006 11

Why a touch screen?Why a touch screen?► Children with autism typically have difficulty with Children with autism typically have difficulty with

remote object references. A touch screen allows remote object references. A touch screen allows direct manipulation – a perceptually direct link direct manipulation – a perceptually direct link between the object and its indication.between the object and its indication.

Why pictures?Why pictures?► We are concerned with We are concerned with

narrative comprehension narrative comprehension not ‘literacy’ or reading not ‘literacy’ or reading ability.ability.

Why a Why a game?game?

► Enjoyment and a Enjoyment and a playful context is playful context is central to our central to our approach.approach.

ICCHP 2006 12

Creating proto-narrativesCreating proto-narratives

►Primitives identified asPrimitives identified as Character Character Background or setting Background or setting SequenceSequence

►Reversible Reversible ►Temporal (may have causality, or be episodic) Temporal (may have causality, or be episodic) ►Narrative (involves motivations of characters)Narrative (involves motivations of characters)

►t-stories (picture narratives and proto-t-stories (picture narratives and proto-narratives collectively) were preparednarratives collectively) were prepared

ICCHP 2006 13

TouchStory displaying a t-story TouchStory displaying a t-story from the category ‘reversible from the category ‘reversible

sequence’sequence’

The game is to select the correct answer from the lower row and drag the picture to the gap in the upper row

ICCHP 2006 14

More example t-stories

character sequencetype c

reversible sequencetype rs

ICCHP 2006 15

temporal sequencetype ts

simple narrative sequence type ns

ICCHP 2006 16

LLongitudinal studyongitudinal study► 12 children (10 ASD) ages 5 - 12 children (10 ASD) ages 5 -

11 years 11 years ► series of 12 visitsseries of 12 visits► 50 sequences used; simple 50 sequences used; simple

graphics, photos, picture story graphics, photos, picture story books books

► about 12 t-stories / child / visit, about 12 t-stories / child / visit, replacement rate about 20% replacement rate about 20%

► later visits, the sequences later visits, the sequences seen were tailored to each seen were tailored to each individual child using an individual child using an adaptive formulaadaptive formula

Children given help or autonomy as appropriate to the child.

ICCHP 2006 17

Study PhasesStudy PhasesPhase 1: information gathering phasePhase 1: information gathering phase

visit 1 or 2 picture-story book narrative visit 1 or 2 picture-story book narrative comprehension taskcomprehension task

visit 1 and 3 -7 using TouchStory, no individual visit 1 and 3 -7 using TouchStory, no individual adaptationadaptation

Phase 2: adaptive phasePhase 2: adaptive phase visits 8 -12 using TouchStory with the t-story visits 8 -12 using TouchStory with the t-story

set adapted to the individual child set adapted to the individual child (visit 11 or 12 final story-book task)(visit 11 or 12 final story-book task)

ICCHP 2006 18

AdaptationAdaptation► Uses a simple adaptive formula Uses a simple adaptive formula

the issue is to identify what this the issue is to identify what this particular child finds difficultparticular child finds difficult

a predefined ordering of difficulty a predefined ordering of difficulty cannot be assumedcannot be assumed

► Aim of adaptation is to provide more Aim of adaptation is to provide more exposure to ‘difficult’ t-stories, providing exposure to ‘difficult’ t-stories, providing a challenging but still enjoyable gamea challenging but still enjoyable game

ICCHP 2006 19

ExpectationsExpectations

►Levelling out of performance during Levelling out of performance during phase 1phase 1

►Initial decline inInitial decline in overall overall performance performance during phase 2 as higher proportion of during phase 2 as higher proportion of difficult t-stories are introduceddifficult t-stories are introduced

►Followed by improvement to above Followed by improvement to above previous level if learning occursprevious level if learning occurs

ICCHP 2006 20

Initial observations (1)Initial observations (1)

ch8 ch7 ch5 ch12 ch11 ch3 ch2 ch4 ch10 ch6 ch9 ch110

20

30

40

50

60

Link

age

Dis

tanc

e

0

0.2

0.4

0.6

0.8

1

1 2 3 4 5 6 7 8 9

pro

po

rtio

n c

orr

ect

Child ch1

e.g

Children in this group

have no trouble with character and background sequences

do least well with TEMPORAL sequences

have narrative comprehension task profiles in the range 3 – 6.25 out of a possible 20

ICCHP 2006 21

Initial observations (2)Initial observations (2)

ch8 ch7 ch5 ch12 ch11 ch3 ch2 ch4 ch10 ch6 ch9 ch110

20

30

40

50

60

Link

age

Dis

tanc

e

0

0.2

0.4

0.6

0.8

1

1 2 3 4 5 6 7 8 9

prop

ortio

n co

rrec

t

Child ch3

e.g

Children in this group• have a less differentiated profile• are most successful with character and background • do least well with NARRATIVE sequences• have narrative comprehension task profiles in the range 0.5 – 2.5 out of a possible 20• children ch2 and ch3 do not have autism

ICCHP 2006 22

Initial observations (3)Initial observations (3)

ch8 ch7 ch5 ch12 ch11 ch3 ch2 ch4 ch10 ch6 ch9 ch110

20

30

40

50

60

Link

age

Dis

tanc

e

Child ch5

e.g

Children in this group• do very well with TouchStory• have narrative comprehension task profiles in the range 8.5 – 18.5 out of a possible 20

0

0.2

0.4

0.6

0.8

1

1 2 3 4 5 6 7 8 9

prop

ortio

n co

rrec

t

ICCHP 2006 23

ConclusionsConclusions

►TouchStory did distinguish among aspects TouchStory did distinguish among aspects of narrative; some aspects of narrative were of narrative; some aspects of narrative were found easier than othersfound easier than others In general the children were more successful choosing In general the children were more successful choosing

characterscharacters, , backgroundsbackgrounds, and , and size of charactersize of character than than with with reversiblereversible, , temporaltemporal or or narrative narrative sequences sequences

The relative difficulty did vary from child to childThe relative difficulty did vary from child to child

ICCHP 2006 24

Relevance to real worldRelevance to real world

► The children still enjoyed TouchStory on the last visit; some The children still enjoyed TouchStory on the last visit; some became increasingly engaged and confident.became increasingly engaged and confident.

► There There isis a significant correlation between the average a significant correlation between the average number of t-stories answered correctly per visit and the number of t-stories answered correctly per visit and the picture-story narrative comprehension task scorespicture-story narrative comprehension task scores

► There was a great deal of difference among the children in There was a great deal of difference among the children in their narrative ability, productive language, and their narrative ability, productive language, and understanding of the taskunderstanding of the task

► No evidence of enhanced understanding of narrative: a No evidence of enhanced understanding of narrative: a longer study needed?longer study needed?

ICCHP 2006 25

What Next?What Next?► New version of TouchStory with New version of TouchStory with

automaticautomatic adaptation adaptation

► Another longitudinal study with Another longitudinal study with longerlonger adaptive phase; does learning take adaptive phase; does learning take placeplace

► Issues includeIssues include the focus and latency of the adaptive the focus and latency of the adaptive

formula, formula, the iconic range and variety of transition the iconic range and variety of transition

types of the proto‑narrativestypes of the proto‑narratives

► Methods of enquiry must remain Methods of enquiry must remain indirect e.g. observation indirect e.g. observation

ICCHP 2006 26

Thank youThank you

►Our goal is ambitious, we do not Our goal is ambitious, we do not expect to provide any quick ‘cure’ expect to provide any quick ‘cure’ for the narrative deficit found in for the narrative deficit found in autismautism

►Even small steps are worthwhile.Even small steps are worthwhile.