PowerPoint Presentation

An innovative approach to reading fluency assessment

Motoli Score An innovative approach to readin1What is MoToLi ScoreA mobile application for assessment of reading fluencyMultiple choice questions based on audio as target word/letter

Motoli score is a mobile application used for the assessment of reading fluency.It uses multiple choice, the target is an audio file, and the student has to choose between 4 possible answer2What is MoToLi Score

Lets what how it works3Results MeasuredAnswers:Number/Percentage of correct answerNumber/Percentage of incorrect or skipped answerNumber of quizzes answered correctly in a minute

The result measured concern accuracy and speed. For the accuracy we measure the percentage of correct answer and for speed we measure the number of quizzed answered correctly in a minute, which is highly correlated with eth number of words read per minute.4Correlation with EgraTo access the correlation of MoToli Score with EGRA a study was conducted in Nicaragua in November 2015.The results were very good, and the application was adjusted to further improve the correlation.To access the correlation of MoToli Score with EGRA a study was conducted in Nicaragua in November 2015.The results were very good.The trial also enabled us to identify a few way to adjust the application in order to improve the correlation

5

MethodologyTeam of assessorsImplementation contextTest designTest procedureResults measuredPopulation testedMethodology: I will present to you the methodology used for the test, the following aspects will be described in details:- Who were the assessors, what was their competency level?- In which context did we implement the test?- How was the test designed?- What was the test procedure?- Which results did we measure?- What population of children did we test?

6Team of assessorsSupervisor: Micaela Gomez trained by RTI had 6 years of EGRA experience.Assessors: Nidia Reyes and Mercedes Aguilar had 4 of EGRA experience.=> Very qualified team.

Team of assessors:Micaela Gomez was supervising and coordinating the assessment. Micaela was trained by RTI during the first EGRA implementation in Nicaragua in 2008. She has been training assessors and running EGRA evaluations for the past 6 years.Nidia Reyes and Mercedes Aguilar were performing the tests. They have both been EGRA enumerators for over 4 years.The team was very qualified to perform the test and to follow the EGRA procedure. We encountered no logistic issues.7Assessors TrainingNidia and Mercedes received a 40 minute training to MoToLi Score.

Assessors Training: Nidia and Mercedes received a 40 minute training to MoToLi Score. They were first taught how to use the tablet and the test as well as a few technical troubleshooting issues. After this short preparation, they simulated the test situation with each other. Lets watch Mercedes and Nidia simulate a testing situation.

8Implementation ContextThe test was performed very poor community, in a school with low reading level

Implementation Context: The test was performed in la Escuala Vera Cruz, a school located in Ciudad Sandino about 30 kilometers north of Managua. This very poor community was selected because of its low results for reading, which were diagnosed during previous EGRA evaluation.

9Implementation ContextThe school was very welcoming, the children behaved very well, and despite a very large number of children in a very small space, there were no visible discipline issues.

10Test DesignEGRA:The original version designed by RTI for Nicaragua in 2008, except that the upper case letters were removed.MoToLi Score:The letter test had 28 quizzes, and the word test 30 quizzes. The 30 target words were taken from the EGRA test, and most of the distractors.Test Design:EGRA:The EGRA version used was the original version designed by RTI for Nicaragua in 2008, except that the upper case letters were removed.MoToLi Score:The letter test had 28 quizzes, and the word test 30 quizzes. Each quiz entailed 1 target word (or letter) and 3 distractors. The 30 target words were taken from the EGRA test, and most of the distractor words were also part of the EGRA evaluation, but some distractors were chosen based on their similarities with the target words.

11Test ProcedureThe assessor introductionEGRA practiceEGRA is performed for letters and then for wordsMoToLi Score practiceMoToLi Score for letters and then wordsTest Procedure: Each assessor performed EGRA and MoToLi for each student. The test procedure was:

The assessor introduces herself and collects information related to the student (Name, sex, age, grade, class).The assessor explains what the test is about and why we are doing the test in order to limit the stress factor.The student does a short EGRA practice.EGRA is performed for letters and then for words.The student does a short MoToLi Score practice.The student performs MoToLi Score for letters and then words.

12Test Procedure

Lets watch a child performing MoToLi Score assessment13Test ProcedureSome students were tested without any assistance

Some students had to retake MoToLi Score because of technical issues or because of a high level of noise. They then took the test on their own without any assistance but with headphones.

Lets watch a child performing the test by him self14EGRA Results MeasuredNumber of letters read correctly per minute Number of words read correctly per minutePercentage of letters read correctlyPercentage of words read correctly

For EGRA, we measured:

Number of letters read correctly per minute using the letter name and not the sound as the correct answer. Number of words read correctly per minute.Percentage of letters read correctly.Percentage of words read correctly.

15MoToLi Results MeasuredNumber of letter quizzes answered correctly per minuteNumber of word quizzes answered correctly per minutePercentage of letter quizzes answered correctlyPercentage of word quizzes answered correctly

For MoToLi, we measured:

Number of letter quizzes answered correctly per minuteNumber of word quizzes answered correctly per minute.Percentage of letter quizzes answered correctly.Percentage of word quizzes answered correctly.

16Population testedKindergarten: 12 students1st grade : 98 students2nd grade: 17 students

Population tested: We mainly wanted to focus on 1st graders, who had just finished their first year of school, and who were therefore expected to be able to read. However we also included some students from kindergarten just with the letter evaluation, and some 2nd graders just with the word evaluation.Kindergarten: 12 students1st grade : 98 students2nd grade: 17 students

17Results Letters FluencyCorrelation ratio: 0.7795% confidence interval [0.65,0.89 ]Result Letters fluency:Correlation ratio: 0.7795% confidence interval [0.65,0.89 ]

18Results Words FluencyCorrelation ratio: 0.8695% confidence interval [0.76,0.95]Result: words fluency:Correlation ratio: 0.8695% confidence interval [0.76,0.95]19Results Letters AccuracyCorrelation ratio: 0.8495% confidence interval [0.74,0.95]Result Letter accuracy: Correlation ratio: 0.8495% confidence interval [0.74,0.95]

20Results Words AccuracyCorrelation ratio: 0.8695% confidence interval [0.76,1]Result words accuracy:Correlation ratio: 0.8695% confidence interval [0.76,1]

21Issues identified during trialTeacher taught letters sound and not the letters nameThe quality of the audio was not perfectThere was a lot of noise outsideIssues identified during trialThe teachers of the Escuela Vera Cruz did not teach the letters name but the letters sound, however the letters sound is considered wrong for the standard EGRA procedure. In one extreme case a girl read only 10 letters correctly per minute but 40 words per minute. However, since the student knew the letters, she had a good result with MoToLi Score, which seems more accurate than EGRA.The quality of the audio was not perfect, partially because the audio files were not recorded locally, and the accent must have been slightly different, only 1 student had a perfect score for letters.There was a lot of noise outside and, because we wanted to be able to observe the child answering, we did not always use headphones. Noise can induce inaccuracy of the evaluation.

22Issue: Noise Level

Lets listen to the level of noise, which can make test conditions very difficult.23Issues identified during trialKids guess rather than skipNon readers could manage with some luck to get a few correct answers using MoToLiVery fast readers did not always try to answer fast on MoToLi Scores quizzesCulturally it seems normal for the kids to guess rather than to skip or say that they dont know. For instance one student while reading the EGRA words recited the name of all the animals she knew.Some students with very low reading skills (close to 0) could manage with some luck to get a few correct answers using MoToLi. A little girl for instance always tapped on the same box. However, we have identified a few ways to limit this occurrence, which will be shared at the end of this report. At the other end of the spectrum, very fast readers did not attempt to answer fast on MoToLi Scores quizzes, unless they were specifically asked to do so. Kids could therefore benefit from a practice test a little longer than the 3 word test which was used in this trial.

24Issue: Luck Factor

Lets watch how this little girl who does not know how to read a word manages to get a luck related score by tapping always on the same box.25MoToLi Score - ImprovementsThe application stops in the following cases:5 consecutive incorrect answers 4 times tapping on the same 40 sec with no correct answerEnable the use of upper case letters

MoToLi Score ImprovementsThe application should stop if the students is a non-reader, just like EGRA stops if no letters/words are read correctly in the first line. We will make the application stop in the following cases:5 consecutive incorrect answers 4 times tapping on the same box 40 sec with no correct answerA new version of MoToLi score will enable the use of upper case letters

26MoToLi Score - New FeaturesComprehension testData collection with synchronization of data with a serverA statistic module

MoToLi Score - New FeaturesWe will add a comprehension test based on multiple choice.The data collection component will be improved to enable synchronization of data with a server.A statistic module will be added to enable a quick overview of the results for a given population.

27Procedure RecommendationsTest the app with about 10 good readersSounds volume as high as possiblePractice with 5 quizzes Make students are aware that time is measured

Procedure RecommendationsWe recommend testing the application with about 10 good readers, to make sure that they can hear the target words correctly.Volume of sound files should be as high as possible.The practice test should include 5 quizzes, in order to acquire the validation gesture. Make sure that the students are aware that they should try to answer as fast as they can.

28Strength of MoToLi Score Result not reliant on enumeratorTraining of an enumerator less than 1 hourAssessment can be performed by anybody Reduces cost of assessmentEasy to administer even in classroom settingCurriculum based assessment Can be tailored for continuous assessment

Strength of MoToLi Score Result not reliant on enumeratorTraining of an enumerator less than 1 hourAssessment can be performed by anybody => Reduces cost of assessmentEasy to administer even in classroom settingCurriculum based assessment => Can be tailored for continuous assessment

29Strength of MoToLi Score Results can be synchronized immediately for analysisResults include geo-localizationMore information availableChildren favor the digital test versus the paper oral testDigital test limits the stress factorChild friendly

Strength of MoToLi Score Results can be synchronized immediately for analysisResults include geo-localization= >More information availableChildren favor the digital test versus the paper oral testDigital test limits the stress factorChild friendly30Possible Use of MoToLi ScoreIn the classroom for continuous assessment, screening and diagnosticProgram evaluations, especially for monitoring of new methodsDistrict or national reading assessment campaignPossible Use of MoToLi ScoreMoToLi Score could be used in the classroom for continuous assessment, screening and diagnostic.MoToLi Score can enable program evaluations, especially when new methods are being tested, and results are needed before a wide scale deployment.District or national reading assessment campaign could benefit from MoToLi Scores easy data collection features.

31

THANK YOU