Dytective: A Game to Detect DyslexiaLuz Rello, Abdullah Ali, Miguel Miguel Ballesteros, Daniela Alarcon Sanchez, Jeffrey P. Bigham

In submission to CHI 2016 (please don’t share)

ABSTRACTMore than 10% of the population has dyslexia, and mostare diagnosed only after they fail in school. This workseeks to change this through scalable early detection viamachine learning models that predict reading and writ-ing difficulties by watching how people interact with alinguistic web-based game: Dytective. The design of Dy-tective is based on (i) the empirical linguistic analysis ofthe errors that people with dyslexia make, (ii) principlesof language acquisition, and (iii) specific linguistic skillsrelated to dyslexia. Experiments with 243 children andadults (95 with diagnosed dyslexia) revealed differencesin how people with dyslexia read and write. We trained amachine learning model that was able to predict dyslexiawith 85.85% accuracy in our dataset. Currently, we areworking with schools to put our approach into practice atscale to reduce school failure as a primary way dyslexiais diagnosed.

Author KeywordsDyslexia; Detection; Screening; Serious Games;Linguistics

ACM Classification KeywordsK.4.2 Computers and Society: Social Issues—Assistivetechnologies for persons with disabilities; K.3 Computersin Education: Computer Uses in Education—Computer-assisted instruction.

INTRODUCTIONMore than 10% of the population has dyslexia. For in-stance, the U.S. Congress reported that from 10 to 17.5%of the population has dyslexia. Dyslexia has a neurologi-cal origin, and results in difficulty with reading and writ-ing [25, 52]. If people know they have dyslexia, they cantrain with effort over time to overcome its negative ef-fects [18]. When people with dyslexia are not diagnosedand provided with appropriate support, they often failin school. For instance, the Spanish Ministry of Educa-tion states that over 40% of the school dropout rate isdue to dyslexia [32]. School failure is a primary way that

Submitted to CHI 2016.

Figure 1. Dytective is a web-based game designed to de-tect dyslexia in an affordable and scalable way. Play-ers complete linguistically motivated activities designedto reveal differences between people with and withoutdyslexia. This photo shows a first stage in which play-ers hear a character that they should click, e.g., and thenclick that letter as many times as possible within a timelimit. Distractor characters are chosen to be particularlydifficult for people with dyslexia to differentiate.

people first learn that they might have dyslexia, whichoften comes too late for effective intervention.

The primary reasons that students are not properly di-agnosed is that current procedures for diagnosis are ex-pensive and require professional oversight [7, 9, 10, 13,35, 41, 38, 48]. Furthermore, practitioners and teacherstell us that children often dislike these tests, which is yetanother reason why children and teachers are unlikely tocomplete them. As a result, even though we know howto detect and diagnose dyslexia, with current approachesmost children will be diagnosed late. Our goal is for ev-eryone to know as early as possible if they might havedyslexia. Our approach to achieving this goal is to makeit easy, inexpensive, and, even, enjoyable to find out.

Our method started with a large corpus of errors madeby people with dyslexia in reading and writing tasks.We then created game activities with progressing levelsof difficulty by leveraging theory of linguistic and vi-sual tasks that are known to be difficult for people with

1

dyslexia. Finally, we packaged these activities into aweb-based game that people can play called Dytective.

Dytective records a wide variety of human-computer in-teraction measures from people playing the game, e.g.mouse movements, click times, errors. A machine learn-ing model trained on 243 participants (95 with diagnoseddyslexia) is able to correctly determine if a person has(or has not) dyslexia with 85.85% accuracy. A numberof challenges make this prediction difficult: (i) our ac-tivities are based on linguistic exercises, and so may notwork well for children still learning to read, (ii) manypeople with dyslexia have already started to work toovercome the effects of dyslexia and so may perform bet-ter than we expect, and (iii) absolute ground truth is dif-ficult to obtain because even professional diagnoses areimperfect.

DYSLEXIAIn the Diagnostic and Statistical Manual of Mental Dis-orders (DSM-V), dyslexia is listed as a specific learn-ing disorder having a neurological origin [2]. Dyslexia ischaracterized by difficulties with accurate and/or fluentword recognition and by poor spelling. These difficultiestypically present as a deficit in the phonological compo-nent of language that is unexpected in relation to othercognitive abilities. Secondary consequences may includeproblems in reading comprehension and reduced readingexperience that can impede growth of vocabulary andbackground knowledge [25, 27].

Why is Dyslexia Difficult to Detect?Even in the UK, a country that effectively treats dyslexiaas compared with other countries, only 5% of the individ-uals with dyslexia are diagnosed and given appropriatehelp; it is estimated that over 85% of adult illiterateshave dyslexia [11]. Even if research agrees in the neu-rological universality of dyslexia, its manifestations aredifferent across languages, depending on the grade of reg-ularity of the language orthographies [21]. For instance,English has an opaque –or deep– orthography (the re-lationships between letters and sounds are inconsistent)and Spanish has a transparent –or shallow– orthogra-phy with more consistent mappings between letters andsounds [44]. While dyslexia manifestation in languageswith opaque orthographies are related to reading andwriting performance, the manifestations of dyslexia inlanguages with shallow orthographies are not that evi-dent, with reading speed and fluency the main predictors[43, 49]. While in an English speaking country a childthat read slower but accurate might not been diagnosedwith dyslexia; in a Spanish speaking country it could bediagnosed as dyslexic. In fact, dyslexia has been calleda hidden disability due to the difficulty of its diagnosisin languages with shallow orthographies [51].

RELATED WORKDiagnosing dyslexia and early detection of risk ofdyslexia have been addressed from different fields, in-cluding psychology, neuroimaging and human-computerinteraction.

Detecting Dyslexia with the Help of a ProfessionalIn psychology, traditional paper-based diagnosis ofdyslexia [9, 10, 13, 35, 38], such as the Oral and Writ-ten Language Scales (OWLS) for English [7] or the Testof Reading and Writing Analysis, ‘Test de Analisis deLectoescritura (TALE)’ [48] for Spanish, analyzes bothreading and writing skills. Diagnoses of dyslexia are con-firmed when the reading and the spelling performanceof the child is significantly under the level expected dueto her or his age and general intelligence.1 We couldnot find a ground truth on the accuracy of these diag-noses, in part because they cannot be disclosed due toconfidentiality requirements. In practice, we are awareof self reported cases where children with dyslexia werewrongly diagnosed specially with disorders that are co-morbid with dyslexia such as attention deficit disorder.2

Dyslexia is related to brain function and can be detectedusing neuroimaging. Recently, neuroimaging with chil-dren with dyslexia has revealed relationships betweenbrain responses at infancy and later reading perfor-mance. Molfese [35] reported that there are brain re-sponses (event-related brain potentials) to speech soundswithin 36 hours of birth that can be used to discriminatechildren who would become readers with dyslexia with8 years old. The accuracy of this prediction is 81%, butit is prohibitively expensive.

Regarding human-computer interaction, the only ma-chine learning approach we are aware of is the one byRello and Ballesteros [41] who used eye-tracking mea-sures coming from 97 subjects (48 with dyslexia) to pre-dict readers with dyslexia. The prediction accuracy ofthis approach is 80.18% by using the most informativefeatures (reading time, mean of fixations and age of theparticipant). Dytective improves on this work not onlyin its performance, but also because it does not requirethe use of an eye-tracker, which is a barrier to use inschools since it is expensive and requires supervision.

Attempts at Early IdentificationLyytinen et al. [29] created the computer game Literate,later called GraphoGame [28],3 to identify children atrisk of having dyslexia before school age in Finland. Itsexercises are aimed towards the connection of graphemes(letters) and phonemes (sounds). They conducted twouser studies with 12 and 41 children between 6 and 7years old with promising results. The authors provide

1Other causes, like insufficient vision and hearing ability,brain injury or lack of opportunity to learn, have to be ex-cluded first [10].2Comorbidity indicates a medical condition, existing simul-taneously, but independently, with another condition.3https://graphogame.com/

2

statistical differences between populations but they donot run any machine learning prediction model. In com-parison Dytective aims at people from all ages startingfrom 7 years old and cover a wider spectrum of cognitiveskills including different levels of language and atten-tional abilities.

There are three projects on developing games to predictdyslexia in pre-readers that, to the best of our knowl-edge, did not report any prediction results. First, Gaggiet al. [20] tested a game with 24 pre-schoolers in Italy,that aimed at eye-hand coordination, visual spatial at-tention, rapid speech-sound identification and discrimi-nation as well as visual-to-speech sound. Second, Vanden Audenaeren et al. [50] performed a user study with20 pre-schoolers in Flanders and are currently developingthe game DYSL-X for early risk detection of dyslexia,which includes letter and end-phoneme recognition aswell as psycho-acoustical tests. Similar to previous ap-proaches, another mobile game is being currently devel-oped at its initial phase in Malaysia by Mohtaram etal. [34]. Again, Dytective differs from these approacheson its wider coverage of the cognitive abilities, on thedependent measures we use, that are totally novel foridentifying dyslexia, and in the fact that we actually re-port prediction accuracy results.

Why is Dytective Different?Dytective is not an game adaptation of current diagnosesand it does not require the help of specialists to predictdyslexia. Dytective differs from prior approaches in itscontent design and prediction model. First, the contentof Dytective are exercises based on (i) the empirical lin-guistic analyses of the errors that people with dyslexiamake, (ii) principles of language acquisition, and (iii)specific linguistic skills related to dyslexia. Second, thisfirst game to use human-computer interaction measuresto train a machine learning model to predict dyslexia.That is, to the best of our knowledge, Dytective is thefirst game that aims at screening dyslexia in Spanishapplying machine learning to measures extracted fromlinguistic and attentional exercises designed on the basisof generated content by people with dyslexia.

METHOD: DYTECTIVE GAME DESIGNDytective is designed for dyslexia detection. It targetslinguistic and attention abilities associated with hav-ing dyslexia. Players proceed through a series of timedstages composed of linguistic exercises of increasing dif-ficulty. Detection is possible when people with dyslexiaperform differently than those without dyslexia. To in-crease the likelihood of this happening, we designed theexercises using a corpus of real errors produced by peo-ple with dyslexia that we collected. Therefore, we expectthat people with dyslexia will make more errors, and thusour method will be able to differentiate.

The goal for Dytective players is to solve as many lin-guistic problems as possible within a time limit. Forinstance, in Stage 1, players hear a letter and then click

Figure 2. Dytective screenshots of first (left) and last(right) exercises of Stage 1, illustrating one way that dif-ficulty is increased progressively.

it as many times as possible within the time limit (Figure2). After each click, the board changes, and the playerreceives points if they chose the correct option.

Content DesignDytective has 17 stages, split into 32 levels consisting of212 exercises in total. Each stage targets a different lin-guistic or attentional skills and all together aim to coverthe maximum potential indicators of dyslexia that couldbe catched via a computer-game. The exercises are in-tended for 7 year old and older. However, for each stage,the exercises are ranked by difficulty, and, in the smallamount of time that is given to the player, older partic-ipants with more advanced language skills are expectedto solve more linguistic problems than younger partici-pants.

Errors as Source of KnowledgeMost approaches to diagnosis are based on reading andwriting measures such as the the number of words readper minute or comprehension of written material. Dytec-tive uses linguistic and attentional exercises designed todistinguish populations with and without dyslexia. Em-pirical analyses show that errors made by people withdyslexia are different from the errors made by peoplewithout dyslexia and that they reflect their difficulties[46]. Hence, the criteria for the linguistic exercises werebuilt on the basis of an analysis of an existing resourceof errors made by people with dyslexia [40].

We analysed the errors from a visual point of view(shapes and visual features shared by the letters of otherlinguistic segments involved in the errors) and from alinguistic point of view, taking into consideration all thelanguage levels that were involved in the errors, mainly,phonetic, phonologically, morphologically and syntacticlevels. The most frequent linguistic and visual featuresshared in the errors were incorporated into the exercises,as described below.

3

Targeting Linguistic and Attentional SkillsWe manually created different kinds of exercises to coverthe maximum number of linguistic and attentional abili-ties related to dyslexia and whose performance can becatched via a computer-based game. In Table 1 wepresent the cognitive skills that each exercise type targets(all the exercises can be grouped in 17 types of exercisesor stages). Most of the exercises address phonologicalawareness because that meta-cognitive skill is the pri-mary factor in solving reading and spelling problems,also in the case of dyslexia [5, 25]. Indirectly, all theexercises target visual attention skills which have beenfound to have a causal relationship with reading acqui-sition in the case of dyslexia [12, 14].

Linguistic Content and Ranking CriteriaThe Dytective exercises get progressively more difficultboth in later stages and within each stage [17]. Eachstage is composed of a number of exercises, ranging fromfour to twenty-six exercises and they are ranked by theirdifficulty. We selected the linguistic input of the ex-ercises, e.g., letters, syllables, etc., using both linguisticpatterns extracted from our error analysis, and the orderin which the linguistic elements and structures are nat-urally acquired [37]. Thus, exercises that appear earliershould be those that are easiest for people with dyslexiato complete, and should also be those that are easiestfor younger children to complete.

In higher difficulty levels, the target letter, syllable, orword(s) tends to be less frequent, longer, has a more com-plex morphology, and has a higher phonetic and ortho-graphic similarity with other words. Both, error analysesand previous literature show that phonetic and ortho-graphic similarity makes it more challenging for peoplewith dyslexia as together with frequency, length, mor-phological complexity [31, 42].

- In Stage 1 the player hears a letter and needs to mapit to its visual representation, which involves Ortho-graphic Processing. The first exercises target vowels,consonants are added. Exercise complexity is addedgradually, for instance the first exercises include vowelsand the last ones contain consonants that are acquiredthe latest in language. Higher levels gradually includedistractors4 that share more phonetic features or vi-sual features with the target letter (Figure 2, right).

For instance, in the first exercise an arbitrary vowelcan act as a target as well as distractors (Figure 2,left); the middle exercises include vowels and conso-nants with that are visually similar (mirror letters)such as a/e, n/u, m/w, and in the most complicatedexercises only consonants that share phonetically andvisual features are taken into account, such as the con-sonants d, b, p, q that share phonetic features (occlusiveconsonants) and visual features (mirror and rotationfeatures) (Figure 2, right).

4Distractors are incorrect options in a multiple-choice ques-tion, designed to resemble the correct answer [33].

- In Stage 2 the player needs to recognize letters bysound (not letter names). The criteria regarding thetarget and distractors selection is similar to Stage 1giving more height to phonological features since thegoal of this exercises is the phoneme to grapheme map-ping, that is sound to letter aiming at the PhonologicalAwareness. In higher levels, targets and distractorsshare more phonetic features, for instance, the soundn (target) is similar to the distractors m, n (voiced andnasal sounds).

- In Stage 3 the player needs to map the syllable theyhear with what they see. The structural syllable com-plexity is gradually increased in the exercises, startingfrom direct syllable (CV)5 such as ba, inverse syllable(VC) i.e. on, checked syllables with CCV structure,i.e. pla and checked syllables with CCVC structure,i.e. glis.

- In Stages 4 and 5 the player needs to recognize wordsand non-words, respectively.6 The target words andthe distractors were selected using the general crite-ria explained above. In these stages we made usedof neighbours or minimal pairs (words with the samelength as the target word which differ in only one let-ter). The higher the ranking the more phoneticallyand orthographically similar the words and non-wordsare.

- In Stage 6 players are presented with a number ofletters and need to spot the one that is different (Vi-sual Attention). Distractors gradually become morephonetically and orthographically similar (Table 1).

- In Stages 7 to 12 players must produce correct wordsby fixing errors based on the real errors that peoplewith dyslexia make. These exercises target Phonologi-cal Awareness at a lexical (word) level. They were de-signed based on the type of errors that appear in textswritten by people with dyslexia, such as addition ofletters, *arround (around); omission of letters, *emty(empty); substitution of letters, *scholl (school); trans-position of letters, *littel (little); and word boundaryerrors such as split words, *mis understanding (mis-understanding), and run-ons, *alot (a lot). Dependingon the fixing operation the exercises are grouped infollowing stages.

- Stage 7: Insertion of a letter.

- Stage 8: Substitution of a letter.

- Stage 9: Reordering letters.

- Stage 10: Reordering syllables.

- Stage 11: Sentence segmentation, that is,split a string of characters into words.

- Stage 12: Deletion of a letter.

5C refers to consonant and V refers to vowel.6A non-word is a word that has no meaning, is not knownto exist, but from a structural point of view it could exist inlanguage.

4

ExerciseType

CognitiveSkill

ExampleExerciseType

CognitiveSkill

Example

Stage 1Letterrecognitionby name

OrthographicProcessing

Stage 2Letterrecognitionby sound

PhonologicalAwareness

Stage 3Syllablerecognitionby sound

PhonologicalAwareness

Stage 4Wordrecognitionby sound

WordRecognition

Stage 5Non-wordrecognitionby sound

PhonologicalMemory

Stage 6Letterdifferentiation

VisualAttention

Stage 7Insertionof letter

PhonologicalAwareness

Stage 8Substitutionof letter

ErrorCorrection &PhonologicalAwareness

Stage 9Letterordering

PhonemicSegmentation&PhonologicalAwareness

Stage 10Syllableordering

SyllabicSegmentation&PhonologicalAwareness

Stage 11Sentencesegmentation

WordRecognition

Stage 12Deletionof letter

PhonologicalAwareness

Stage 13Errordetection

SyntaxicAwareness

Stage 14Errordetection

SemanticAwareness

Stage 15Lettersequencememorization

VisualMemory &WorkingMemory

Stage 16Worddictation

Word Writing

Stage 17Non-worddictation

Non-WordWriting &PhonologicalMemory

Table 1. Linguistic and attentional abilities targeted by each stage in Dytective. The 17 stages are split into 32 levelsconsisting of 212 exercises in total.

5

Whenever possible, we used word frequency to makeexercises more challenging. Frequent words are rec-ognized significantly faster by everybody, whereas lessfrequent words are especially challenging to recognizefor people with dyslexia [24, 42]. Therefore, we usedmore frequent words as distractors. For example, inthe “insert a letter exercise” (Table 1, Stage 8) dis-tractors for the string lu es were s, c, m, z, u. Manyof the participants failed and chose m in an attemptto create the word lunes, ‘Monday’ which is 5 timesmore frequent than the correct word luces, ‘lights’.7

- At Stages 13 and 14 the player needs to spot writ-ten errors in sentences. The reason why we includedthese exercises is because one of the main challengesthat people with dyslexia face is that they do not con-sciously detect errors while reading [4]. We used real-word errors (correctly spelled words that are not theone the user intended to write, i.e. a letter *formyou instead of i.e. a letter from you). In Stage 13the errors occur in lexical words8 and in Stage 14 infunction words.9 This way each of the group of ex-ercises aim at different comprehension language levelsand linguistic skills, Syntaxic Awareness and SemanticAwareness, respectively. We also made this differenti-ation because lexical and function words are processeddifferently [37].

For these stages we also took into consideration thefrequency of the n-grams (group of words) within thesentences, under the assumption that errors occurringin more frequent n-grams are more difficult to spot be-cause frequency is related with word recognition speed.For instance, the real world error Expect to be stopped*buy an agent is more difficult to spot than errors oc-curring in less frequent n-grams such as These items*mite interfere with the navigational equipment.

- At Stage 15 the user needs to memorize sequencesof letters with increasing difficulty (Visual Memory &Working Memory). That is, increasingly the sequenceto remember contain letters that are less frequent, or-thographically less transparent,10 and share visual andphonetic features among each other.

7Frequency was calculated using the hit counts of the occur-rences of the word in all the webpages written in Spanish.https://www.google.com/advanced search.8Lexical words are content words, i.e. nouns, verbs, adjec-tives and most adverbs. They have a lexical meaning in con-trast with the grammatical meanings expressed by functionwords, such as prepositions or conjunctions.9Function words are words that have little lexical meaning,but instead serve to express grammatical relationships withother words within a sentence, such prepositions, pronouns,or conjunctions.

10Letter whose sound correspondence is not straightforward,that is, letters that can correspond to different sounds de-pending on the context, for instance, letter c can be pro-nounced as /k/ in casa, ‘house’ or as /Tff/ in cereza, ‘cherry’.

- Finally, at Stages 16 and 17 measure the writingperformance and the Phonological Memory via Wordand Non-Word Dictation. The criteria to select thewords and the non-words are same for the rest of theStages. For instance, the first exercises start with lex-ically simpler non-words, i.e. tada while in the higherlevels the player is asked to write mabadana whoseletters are more likely to be mistaken by people withdyslexia according to our empirical analyses of errors,because they share phonological and visual features.

Interface DesignSince text presentation significantly impacts the textreadability of people with dyslexia we used black texton a white background, a large font size (minimum 18points) [3] and the monospaced Courier font face, whichbenefits both populations with and without dyslexia [1].

ImplementationDytective is a web based game written in HTML5,CSS and Javascript with a backend PHP server and adatabase. By using these web technologies it is possibleto play Dytective on different devices such as desktops,tablets, and mobile phones. It was implemented with ahigh level abstraction to make it easily portable to nativeiOS or Android application for future implementations.

User TestingTo ensure that the exercises were appropriate for theplayer ages, we carried out a set of pilot tests. Eachstage was performed by two students starting from 6to 15 years old. These pilots allowed us to rank theexercises and adjust the exercises according the childrenabilities. First, we observed if the participant could makethe exercise and second we asked the participant to rankeach exercises as easy, medium or hard. On the basis ofthese pilots we redefined the content of the first exercisesof each stage to be suitable for young children.

To ensure usability and user engagement, we iteratedthe application in three series of pilot tests with the samechildren, using the think-aloud method [26]. We replacedthe written explanations of the exercises by oral instruc-tions. Also, to increase engagement, we added pointsand encouraging voices every time they complete a stage.The pilots also helped us to define a fixed time from eachstage, ranging from 30 to 60 seconds. The fixed time al-low us to ensure that all players with different linguisticskills can access to a minimum and maximum amount ofexercises; also we added a counter to the remaining timeleft because it engaged the participants.

Whenever it was possible (in seven stages) we includeda Whac-A-Mole interaction design because it engagedchildren. Finally, to show the increasing of difficulty wealso established levels based on the number of cells inthe grid starting with a 3x3 grid up to 6x6 (Figure 2,left and right).

6

EXPERIMENTAL STUDYWe conducted a study with 243 participants (95 withdiagnoses) using a within-subject design. The goal ofthe study was to collect data needed to run a machinelearning experiment to find out if error-based linguisticproblems can predict dyslexia in Spanish. All of the par-ticipants played all stages of Dytective over 15 minutes,but may not have advanced through all of the exercisesin each stage.

ParticipantsParticipants were recruited through a public call thatdyslexia associations distributed to their members.11 95participants had a confirmed diagnosis of dyslexia in-cluding the date the the place where they were diag-nosed; 31 were at risk of having dyslexia or suspectedthat have dyslexia.12

The participants with dyslexia or at risk of havingdyslexia consisted of 126 people (68 female, 58 male).Their ages ranged from 7 to 68 (M = 20.09, SD =14.47). The group of participants without dyslexia wascomposed of 117 people (74 female, 43 male). Their agesranged from 7 to 70 (M = 18.62, SD = 11.94).

All the participants’ first language was Spanish, al-though 18 with dyslexia and 10 without dyslexia werefully bilingual in Spanish and another language spokenin the area where they live, such as Basque, Catalan,and Galician. 74 participants with dyslexia had previ-ously failed Spanish subject at school, compared to only30 participants without dyslexia.

Dependent MeasuresTo quantify performance, we used the following depen-dent measures extracted for each group of exercises.

i Number of Clicks per stage.

ii Hits, i.e. number correct answers.

iii Misses, i.e. number in correct answers.

iv Score i.e. sum of correct answers per group of ex-ercises.

v Accuracy defined as the number of Clicks dividedby the number of Hits.

vi Missrate defined as the number of Clicks dividedby the number of Misses.

Materials and ProcedureWe sent an announcement of the study to the main as-sociations of dyslexia of hispanic countries and countrieswith large Spanish speaking populations, mainly Ar-gentina, Chile, Mexico, Spain and the USA. We also sent

11Countries and associations are anonymized: 8 state associ-ations, 2 national associations and one international associa-tion of dyslexia.

12Except for 17 adults, all were under observation by profes-sionals, the step before having an official diagnosis.

the call to specialized centers that support people withdyslexia. Interested potential participants contactedus, and after we checked the participation requirements(age, mother languages and technical requirements) weset up a date to supervise the study. We met with theparticipants (and their parents in case the participantwas underage) online or by telephone. After they signedthe online consent and/or parental consent we gave themspecific instructions and they completed the study. Par-ents were specifically warned that they could not helpedtheir children to play Dytective and were asked again af-terwards to double check. The data of two participantswith dyslexia had to be deleted because their mothersexplained to us that helped their children. One schooland one specialized center collaborated in the study. Forthese cases the parental consent was obtained in advanceand the study was supervised by the school counselorand the therapist respectively. We deliberately carriedout the study in three different settings (home, schooland a specialized centre) so our results are independentof the these settings.

DATASETOur dataset was derived from the experimental studypresented in the previous section with 243 participants(95 with diagnosed dyslexia). The dataset is composed of197 features per participant, that is, 47,871 data points.From the dataset we extracted the following features,marked as D if the participant has dyslexia, N if not,and M (maybe) if the participant suspects that he orshe has dyslexia but is not diagnosed.

[1] Age of the participant ranging from 7 to 70years old.

[2] Gender of the participant, a binary featurewith two values female and male.

[3] Second mother language in case of bilingualism;all the participants had Spanish as mother language.

[4] Spanish subject. This is a binary feature withtwo values, yes when the participant has ever failedSpanish subject at school and no when the participanthave never failed that subject among all the schoolhistory.

[5-197] Performance measures. These featurescorrespond with the six dependent measures (Clicks,Hits, Misses, Score, Accuracy, and Missrate) we gatherper level played (32 levels), that is, 192 performancefeatures corresponding to different cognitive skills (seeTable 1).

Some of the features have numeric (real or integer) val-ues, so we established ranges for each of them to dis-cretize the data by the population median.

7

RESULTSIn order to find out whether it is feasible to detect peo-ple with dyslexia after playing Dytective, we set up amachine learning experiment. Machine learning is thescientific discipline that studies algorithms that can learnfrom data and make predictions. The output of a ma-chine learning algorithm is called a model, which is capa-ble of making predictions given unseen data. In this case,the goal is to predict whether someone has dyslexia ornot based on the data collected while participants playedDytective.

We used the binary classifier LIBSVM [8] in the poly-nomial Support Vector Machine (SVM) set-up. A SVMis a method for supervised learning that analyzes dataand recognizes patterns for classification. Given a set oftraining examples, each marked as belonging to a cate-gory (in our case either having dyslexia or not), an SVMtraining algorithm builds a model that assigns new ex-amples into the categories. When there is an input forthe classifier it tries to assign a category to the inputand then this is the classification output. Our SVM istrained on datasets like the one described in the DatasetSection, and it is able to perform predictions on newparticipants that may play Dytective.

We performed a cross validation experiment by dividingthe dataset in 243 different subsets having only one par-ticipant each. We then iteratively trained a statisticalmodel on all the data but one participant (242 partic-ipants) and tested the one held out. At the end, wehad all the data tested independently. The participantsmarked as M (maybe) are used for training the modelsas if they are D (participants with dyslexia) but theyare not used for evaluation. This means that we have212 participants to test, and we train each model, a to-tal of 212 models, with 242 experiments performed byparticipants.

The initial results suggest that the model is able to pre-dict people for having or not having dyslexia quite accu-rately with a final result of 81.60% in the cross-validationexperiment by using all features (151 performance fea-tures extracted from the game, plus age, gender, motherlanguage and school performance), meaning that the sta-tistical models are able to make a correct prediction in173 of the 212 participants.

Optimization: Feature SelectionIn order to improve performance, we carried out a featureselection experiment following a backward algorithm.We start testing a model with all features, and we it-eratively removed features one by one by training newmodels; if the performance was better or equal than be-fore we permanently remove the feature from the featureset, seeking more informative features.

After we have a partially reduced the feature set, we car-ried out a redundancy selection experiment, in which weremoved features in pairs, by testing all possible combi-nations in a double loop, meaning that we fix a partic-

ular feature and we start removing all feature plus theparticular feature that we have fixed before. If the per-formance is better or equal than before we remove thepair of features from the feature set.

After the optimization round we obtained an improvedresult of 85.85%, which increases the previous score sub-stantially and reduces the number of features, from 198to 150. Some of the dependent measures from someStages were left out, such as the number of Clicks inStage 4. The model selected features from all the stages.The model is now capable of correctly predicting thecondition of 182 of the 212 participants.

DISCUSSIONOur machine learning model trained on data from Dy-tective is able to classify people as having dyslexia ornot with fairly high accuracy. In this section, we explorethe utility of the different features used and discuss theerrors made by the model.

Feature AnalysisTables 2 and 3 depict the information gain of each of thefeatures for each stage. Table 2 shows the result of themodel when we only use features from a single stage, andTable 3 shows the results of the model when we use allthe selected features except for the features of a particu-lar stage. The main conclusion to extract is that all thestages are needed; the tasks and the cognitive skills thattarget each stage complement each other aiming at thedifficulties –linguistic and attentional– that people withdyslexia have.

As shown in Table 2, the most informative stage by itselfis the Stage 16 (Word dictation), achieving an accuracyof 75.94, by only using the performance measures ex-tracted from that particular stage. However, as shownin Table 3 when using the rest of the features, Stage 16is more redundant than others being the Stage 2 (Let-ter recognition by sound) the most relevant in order toachieve a higher performance when all the stages are tak-ing into account. When we remove the features from theStage 2 and we run the experiment with the rest of thestages, the model only achieves 74.06.

One possible explanation for why all the stages are infor-mative could be that all indirectly –or directly in Stage6 (Letter differentiation)– aim at visual attention. Pre-vious studies have found that visual-spatial attentiondeficits are an indicator of dyslexia [12, 19, 36]. In fact, astudy done with 96 pre-reader Italian-speaking children(5 years old) show that performance on visual searchtasks is related to later reading acquisition [14].

Another reason why all the stages are informative couldbe that all of them integrate the factors of word fre-quency and neighborhood frequency. Previous studieshave shown that low frequency words and the existenceof neighborhood with high frequency have an effect onsome tasks such as reading and visual search [39] or lex-ical decision and naming [22]. The are some studies that

8

Dataset Accuracy

Only Stage 1 72.17% (153/212)Only Stage 2 70.75% (150/212)Only Stage 3 65.09% (138/212)Only Stage 4 67.92% (144/212)Only Stage 5 69.34% (147/212)Only Stage 6 61.79% (131/212)Only Stage 7 71.70% (152/212)Only Stage 8 74.06% (157/212)Only Stage 9 74.53% (158/212)Only Stage 10 72.17% (153/212)Only Stage 11 72.17% (153/212)Only Stage 12 64.62% (137/212)Only Stage 13 68.40% (145/212)Only Stage 14 63.68% (135/212)Only Stage 15 73.11% (155/212)Only Stage 16 75.94% (161/212)Only Stage 17 73.11% (155/212)

Table 2. Accuracy of the classifiers in the cross validationexperiment, using the optimized feature set and only thefeatures from a particular stage. Each stage was able toperform much better than chance even without the otherstages.

Dataset Accuracy

All without Stage 1 75.94% (161/212)All without Stage 2 74.06% (157/212)All without Stage 3 76.42% (162/212)All without Stage 4 76.42% (162/212)All without Stage 5 78.30% (166/212)All without Stage 6 79.25% (168/212)All without Stage 7 76.42% (162/212)All without Stage 8 76.42% (162/212)All without Stage 9 76.89% (163/212)All without Stage 10 79.72% (169/212)All without Stage 11 77.36% (164/212)All without Stage 12 77.83% (165/212)All without Stage 13 78.77% (167/212)All without Stage 14 81.13% (172/212)All without Stage 15 81.60% (173/212)All without Stage 16 79.72% (169/212)All without Stage 17 83.49% (177/212)

All 85.85% (182/212)

Table 3. Accuracy of the classifiers in the cross valida-tion experiment, using the optimized feature set and theablated conditions in which all the features from a par-ticular Stage are removed. The last row shows the resultin which all the stages are included. Every feature wasnecessary to achieve the highest accuracy rate.

suggest that the effects of word frequency in readingspeed [24] and neighborhood density in lexical decition[30] could be even stronger in people with dyslexia.

It was surprising that the feature of ‘age’ was not selectedin the set of the most informative features. This is sur-prising because especially on children reading speed iscorrelated to age. Also, the age feature was a highly

Table 1

Performace 86.20 84.21 97.05 82.35 83.33 66.66

Age 10 15 25 35 45 more 45

0.00

25.00

50.00

75.00

100.00

Performace Age

0.00

50.00

100.00Prediction Accuracy (%)

66.66 (6/9)

83.33 (15/18)

82.35 (14/17)

97.06 (33/34)84.21

(64/76)86.20 (50/58)

7-10 11-15 16-25 26-35 36-45 <46 years old 0.00

50.00

100.00

66.66 (6/9)

83.33 (15/18)

82.35 (14/17)

97.06 (33/34)84.21

(64/76)86.20

(50/58)

Prediction Accuracy (%)

7-10 11-15 16-25 26-35 36-45 >46 years oldFigure 3. Dytective’s accuracy by age group. Accuracyis highest in the 16-25 age group, perhaps because thisgroup is most likely to have a correct diagnosis but has notyet trained to overcome the effects of dyslexia. Detectionaccuracy appears to go down with older adults, perhapsbecause those in this group with dyslexia have largelyovercome its effects.

,

predictive feature in a previous study that predicteddyslexia applying machine learning to eye-tracking mea-sures, that is different indicators of reading speed such asnumber of fixations or mean of fixations [41]. One possi-ble explanation for our algorithm to be age independentis that we do not directly measure reading speed in ourexercises as most of the current diagnoses of dyslexia do[9, 13, 38, 48]. It is also worth noting that this fact in-dicates that the method is more general and would notincorrectly classify people according to their age.

Even if we do not directly measure reading speed, thatmeasure have been pointed out as the strongest indicatorof dyslexia in languages with shallow orthographies suchas Spanish [43] or Italian [49]. However, some of ourexercises require a little bit of reading (specially Stages12, 13 and 14) so indirectly reading speed is taken intoconsideration via other performance measures.

Error AnalysisWe analyzed the cases that our algorithm classifies par-ticipants wrong, the could be divided into three groups.

First, we found errors in in classifying young childrenwithout dyslexia (7-15 years old). A possible explanationof this is that at that age children are still acquiringbasic reading skills and writing skills [16] and the time ofacquisition of those reading skills varies across subjects[53]. Therefore it is possible that some classified with riskof having dyslexia are just acquiring linguistic skills in aslower pace. We believe this weakness of the algorithmcould be solve with more training data.

Second, we have a group of errors coming from teenagersand college students with dyslexia that are classified andpeople without dyslexia. These cases mainly correspond

9

to people who was diagnosed a long time ago (morethan five years) and that have received intervention sincethem. This is probably because these group have alreadylearn how to overcome dyslexia and have developed theproper compensating strategies. In fact, previous studiesshow that with proper intervention the skills of peoplewith dyslexia can be comparable to the rest [23, 45], andthere was even found evidence of amelioriation in brainregions associated with phonological processing [47].

Third, we observed that the algorithm fails with olderadults (specially starting from 45 years old). For thisgroup there are errors in both directions, people withoutdyslexia are classified as the risk group and vice versa.The possible explanation for this is the frequency of ex-posure to reading that the older participants have be-cause the frequency in with people are exposed to wordshave a direct effect on their reading speed [6]. While dur-ing school years that frequency is moderately homoge-nous due the schooling when it comes to adults it reallydepends on the personal choices and on the type of jobthat the participant has.

FUTURE WORKDytective presents a number of opportunities for futureresearch. The current version only works for people withsome reading ability, which means it cannot be used ef-fectively for very young children. We plan to design aversion of Dytective for pre-readers, starting from 3 to6 years old. This would require the development of ex-ercises that are less dependent on linguistic ability. Forexample, we will design new stages that focus on ele-ments of visual-spatial attention, which have previouslybeen found to correlate with reading acquisition amongyounger age children [14, 19, 36].

Now that we know that Dytective has promise fordyslexia detection, it makes sense to explore whether itslinguistic and visual attention exercises can be used forintervention. We believe that they have potential to helpto overcome dyslexia because (i) training exercises basedon letter recognition have been previously shown to im-prove the performance of children with dyslexia [29], and(ii) the increase of attention abilities, i.e. by playingcomputer action games, has been previously shown tohave a beneficial impact on reading abilities [15].

We are also currently working to conduct a large-scalestudy with schools to investigate Dytective over time.We will use Dytective to detect dyslexia with studentswho have not yet been diagnosed. We will then followthem through their schooling to gauge the effectivenessof Dytective in its intended context.

Finally, we plan to extend Dytective to other languages,such as English and German, and to integrate the gameinto a free web-based platform that it can be easilyshared. If we attract enough people to play the game,then we might be able to estimate the prevalence ofdyslexia among web users, which is a long-standing chal-lenge.

CONCLUSIONWe have presented a game to detect dyslexia called Dy-tective. This game uses linguistic and attentional exer-cises that we designed in order to illustrate differencesbetween people with and without dyslexia. A machinelearning model trained on data collected from 243 Dy-tective players was able to predict dyslexia with 85.85%accuracy. Because Dytective is easy to access on theWeb and does not require special equipment, it is easyto scale. Our hope is that it will lead to earlier detectionof dyslexia and prevent children from being diagnosedwith dyslexia only after they fail in school.

REFERENCES1. Ability Net. 2013. NHS Choices: Help with

accessibility, Changing fonts on your computer.(2013).http://www.nhs.uk/accessibilityhelp/index.html.

2. American Psychiatric Association. 2013. Diagnosticand statistical manual of mental disorders,(DSM-V). American Psychiatric Publishing,Arlington, VA.

3. British Dyslexia Association. 2012. Dyslexia StyleGuide. (January 2012).http://www.bdadyslexia.org.uk/.

4. Maggie Bruck. 1988. The word recognition andspelling of dyslexic children. Reading ResearchQuarterly (1988), 51–69.

5. Maggie Bruck. 1992. Persistence of dyslexics’phonological awareness deficits. Developmentalpsychology 28, 5 (1992), 874.

6. Manuel Carreiras, Andrea Mechelli, and Cathy J.Price. 2006. Effect of word and syllable frequencyon activation during lexical decision and readingaloud. Human brain mapping 27, 12 (2006),963–972.

7. Elizabeth Carrow-Woolfolk. 1995. OWLS, Oral andWritten Language Scales. NCS PearsonIncorporated.

8. Chih-Chung Chang and Chih-Jen Lin. 2011.LIBSVM: A library for support vector machines.ACM Transactions on Intelligent Systems andTechnology 2 (2011), 27:1–27:27. Issue 3. Softwareavailable at http://www.csie.ntu.edu.tw/~cjlin/libsvm.

9. Fernando Cuetos, Blanca Rodrıguez, Elvira Ruano,and David Arribas. PROLEC-R. Baterıa deevaluacion de los procesos lectores, revisada. TEA,Madrid.

10. Horst Dilling. 2011. Internationale Klassifikationpsychischer Storungen: ICD-10 Kapitel V (F) :klinisch-diagnostische Leitlinien (8., uberarb. aufl.unter berucksichtigung der anderungenentsprechend icd-10-gm ed.). Huber, Bern.

11. Dyslexia Research Institute. 2015. Dyslexia,Identification. (January 2015). www.dyslexia-add.org.

10

12. Andrea Facoetti, Pierluigi Paganoni, MassimoTuratto, Valentina Marzola, and Gian GastoneMascetti. 2000. Visual-spatial attention indevelopmental dyslexia. Cortex 36, 1 (2000),109–123.

13. Angela J. Fawcett and Rod L. Nicolson. Test parala deteccion de la dislexia en ninos (DST-J). TEA,Madrid.

14. Sandro Franceschini, Simone Gori, Milena Ruffino,Katia Pedrolli, and Andrea Facoetti. 2012. A causallink between visual spatial attention and readingacquisition. Current Biology 22, 9 (2012), 814–819.

15. Sandro Franceschini, Simone Gori, Milena Ruffino,Simona Viola, Massimo Molteni, and AndreaFacoetti. 2013. Action video games make dyslexicchildren read better. Current Biology (2013).

16. Uta Frith. 1985. Beneath the surface ofdevelopmental dyslexia. Surface dyslexia 32 (1985).

17. Christopher Cunningham Gabe Zichermann. 2011.Gamification by Design: Implementing GameMechanics in Web and Mobile Apps. O’Reilly.

18. John de Gabrieli. 2009. Dyslexia: a new synergybetween education and cognitive neuroscience.Science 325, 5938 (2009), 280–283.

19. John de Gabrieli and Elizabeth S. Norton. 2012.Reading abilities: importance of visual-spatialattention. Current Biology 22, 9 (2012),R298–R299.

20. Ombretta Gaggi, Giorgia Galiazzo, ClaudioPalazzi, Andrea Facoetti, and Sandro Franceschini.2012. A serious game for predicting the risk ofdevelopmental dyslexia in pre-readers children. InProc. ICCCN’12. IEEE, 1–5.

21. Nata Goulandris (Ed.). 2003. Dyslexia in differentlanguages: Cross-linguistic comparisons. WhurrPublishers, London.

22. Jonathan Grainger. 1990. Word frequency andneighborhood frequency effects in lexical decisionand naming. Journal of memory and language 29, 2(1990), 228–244.

23. Beve Hornsby. 1986. Overcoming dyslexia. MartinDunitz, London.

24. Jukka. Hyona and Richard K. Olson. 1995. Eyefixation patterns among dyslexic and normalreaders: Effects of word length and word frequency.Journal of Experimental Psychology: Learning,Memory, and Cognition 21, 6 (1995), 1430.

25. International Dyslexia Association. 2011.Frequently Asked Questions About Dyslexia.(2011). www.interdys.org.

26. Clayton Lewis. 1982. Using the “thinking-aloud”method in cognitive interface design. IBM TJWatson Research Center.

27. G. Reid. Lyon, Sally. E. Shaywitz, and Bennet. A.Shaywitz. 2003. A definition of dyslexia. Annals ofDyslexia 53, 1 (2003), 1–14.

28. Heikki Lyytinen, Jane Erskine, Janne Kujala,Emma Ojanen, and Ulla Richardson. 2009. Insearch of a science-based application: A learningtool for reading acquisition. Scandinavian journalof psychology 50, 6 (2009), 668–675.

29. Heikki Lyytinen, Miia Ronimus, Anne Alanko,Anna-Maija Poikkeus, and Maria Taanila. 2007.Early identification of dyslexia and the use ofcomputer game-based practice to support readingacquisition. Nordic Psychology 59, 2 (2007), 109.

30. Chiara Valeria Marinelli, Daniela Traficante,Pierluigi Zoccolotti, and Cristina Burani. 2013.Orthographic neighborhood-size effects on thereading aloud of Italian children with and withoutdyslexia. Scientific Studies of Reading 17, 5 (2013),333–349.

31. Eva Marinus and Peter F. de Jong. 2010. Size doesnot matter, frequency does: Sensitivity toorthographic neighbors in normal and dyslexicreaders. Journal of experimental child psychology106, 2 (2010), 129–144.

32. Ministerio de Educacion, Cultura y Deporte(Ministry of Education, Culture and Sports). 2012.Informe Redie. La atencion al alumnado condislexia en el sistema educativo en el contexto delas necesidades especıficas de apoyo educativo(REDIE report. Attention to students with dyslexiain the educational system in the context of thespecific educational needs). Gobierno de Espaa(Government of Spain).

33. Ruslan Mitkov, Le An Ha, Andrea Varga, and LuzRello. 2009. Semantic similarity of distractors inmultiple-choice tests: extrinsic evaluation. In Proc.EACL Workshop GeMS ’09. 49–56.

34. Saifuddin Mohtaram, Naim Che Pee, andAbdul Samad Sibgatullah. 2014. Mobile DyslexiaScreening Test: A New Approach through MultipleDeficit Model Mobile Game to ScreenDevelopmental Dyslexia Children. In MalaysiaUniversity Conference Engineering Technology.

35. Dennis L. Molfese. 2000. Predicting dyslexia at 8years of age using neonatal brain responses. Brainand language 72, 3 (2000), 238–245.

36. Kristen Pammer. 2013. Temporal sampling invision and the implications for dyslexia. Frontiersin human neuroscience 7 (2013).

37. Steven Pinker. 2009. Language Learnability andLanguage Development. Harvard University Press.

38. Gonzalez Portal. 1992. Exploracion de lasDificultades Individuales de Lectura (EDIL-1).TEA, Madrid.

11

39. Keith Rayner and Gary E. Raney. 1996. Eyemovement control in reading and visual search:Effects of word frequency. Psychonomic Bulletin &Review 3, 2 (1996), 245–248.

40. Luz Rello, Ricardo Baeza-Yates, and JoaquimLlisterri. 2014. DysList: An Annotated Resource ofDyslexic Errors. In Proc. LREC 2014. Reykjavik,Iceland, 1289–1296.

41. Luz Rello and Miguel Ballesteros. 2015. DetectingReaders with Dyslexia Using Machine Learningwith Eye Tracking Measures. In Proc. W4A ’15.Florence, Italy.

42. Jascha Russeler, Stefan Probst, Sonke Johannes,and Thomas F. Munte. 2003. Recognition memoryfor high-and low-frequency words in adult normaland dyslexic readers: an event-related brainpotential study. Journal of clinical andexperimental neuropsychology 25, 6 (2003),815–829.

43. Francisca Serrano and Sylvia Defior. 2008. Dyslexiaspeed problems in a transparent orthography.Annals of Dyslexia 58, 1 (2008), 81–95.

44. Philip. H. K. Seymour, Mikko Aro, and Jane M.Erskine. 2003. Foundation literacy acquisition inEuropean orthographies. British Journal ofPsychology 94, 2 (2003), 143–174.

45. Sally E. Shaywitz. 2003. Overcoming dyslexia: Anew and complete science-based program for readingproblems at any level. Knopf.

46. Christopher. Sterling, Marion. Farmer, Barbara.Riddick, Steven. Morgan, and Catherine.Matthews. 1998. Adult dyslexic writing. Dyslexia 4,1 (1998), 1–15.

47. Elise Temple, Gayle K. Deutsch, Russell A.Poldrack, Steven L. Miller, Paula Tallal,Michael M. Merzenich, and John de Gabrieli. 2003.Neural deficits in children with dyslexiaameliorated by behavioral remediation: evidencefrom functional MRI. Proceedings of the NationalAcademy of Sciences 100, 5 (2003), 2860–2865.

48. Josep Toro and Montserrat Cervera. 1984. TALE:Test de Analisis de Lectoescritura (TALE: LiteracyAnalysis Test). Visor, Madrid.

49. Patrizio E. Tressoldi, Giacomo Stella, and MarziaFaggella. 2001. The Development of Reading Speedin Italians with Dyslexia A Longitudinal Study.Journal of learning disabilities 34, 5 (2001),414–417.

50. Lieven Van den Audenaeren, Veronique Celis,Vero Vanden Abeele, Luc Geurts, Jelle Husson, PolGhesquiere, Jan Wouters, Leen Loyez, and AnnGoeleven. 2013. DYSL-X: Design of a tablet gamefor early risk detection of dyslexia in preschoolers.In Games for Health. Springer, 257–266.

51. Frank R. Vellutino, Jack M. Fletcher, Margaret J.Snowling, and Donna M. Scanlon. 2004. Specificreading disability (dyslexia): What have we learnedin the past four decades? Journal of ChildPsychology and Psychiatry 45, 1 (2004), 2–40.

52. World Health Organization. 1993. Internationalstatistical classification of diseases, injuries andcauses of death (ICD-10) (10th ed.). World HealthOrganization, Geneva.

53. Johannes C. Ziegler and Usha Goswami. 2005.Reading acquisition, developmental dyslexia, andskilled reading across languages: a psycholinguisticgrain size theory. Psychological bulletin 131, 1(2005), 3.

12