RTI International, Research Triangle Park, North Carolina, USA

National Level Classroom Monitoring:

The Impact of Kenya Tusome on

Learning Outcomes and Accountability

Dr. Benjamin Piper

Senior Director, Africa Education

RTI International

2

Principles for Selecting M&E Tools Using ICT

• Test Big Ideas rigorously

• Test usability consistently

• Plan for termly updates

• Focus on demand creation for the data

• Embed data availability in gov’t systems

• Focus on positive reinforcement

3

4

Curriculum

Support

Officers

Tablets Program Tools

1. Tangerine Tutor

– Cloud-based observational tool

– Uploads tied to reimbursement

2. E-versions of instructional materials

3. Instructional videos

4. Papaya for letter sound practice

5

Tangerine Tutor

6

Tutor Classroom Observation Tool

Tap this button

Tangerine Tutor – 3 Embedded Pupil assessments

8

Tutor - Teacher feedback based on classroom responses

9

10

Tusome Data Dashboard

11

Tusome classroom observation dashboard

National Data

12

County Data

13

14

GPS data

15

Local level data

16

School Level Data

17

Data on Data Usage

• 2016/17 national survey

• 76% of the coach population

• 72% of the county director

population

18

Coach ICT Tool Utilization

19

23.7

42 45.8 42.7

74.350

16.914.5

0

10

20

30

40

50

60

70

80

90

100

Tangerine Tutor PDF of materials Instructional videos Papaya soundpractice app

I Use This Tool During Every Coaching Session…

Agree Strongly agree

Data for Accountability Framework

Level 1 – Data is available

Level 2 – Data is used

Level 3 – Data is used to make changes

20

County Directors Use of Data for Accountability

21

16.4

11.9

28.4

50.7

52.2

65.7

66.9

79.4

89.7

89.7

0 20 40 60 80 100

Data affected instructional materials

Data affected resource allocation

Data affected decision to reassign orfire coaches

Gave positive feedback to coaches

Gave negative feedback to coaches

Data affected supervisory visits

Received data via email

Coaching data affected your county

Issued a tablet

Trained on Tusome

Data is

Available

Data is

Used

Data

Makes

Changes

22

Tusome’s Impact on English Benchmarks

23

Tusome External Evaluation Effect Sizes (SD)

24

0.670.63

1.04

0.76

0

0.2

0.4

0.6

0.8

1

1.2

English Kiswahili

Grade 1 Grade 2

Freudenberger & Davis (2017)

Reflections

• Test the M&E system before roll out

• Focus on classroom support

• Get data to decision-makers

• Build demand for accountability data

• Find simple ways to access data

• The point - increase take-up and utilization

25

26

Thank you!

bpiper@rti.org

Monitoring Materials Utilization

27

[CELLRANGE][CELLRANGE]

[CELLRANGE][CELLRANGE]

[CELLRANGE]

[CELLRANGE][CELLRANGE]

[CELLRANGE] [CELLRANGE][CELLRANGE]

[CELLRANGE]

[CELLRANGE]

[CELLRANGE][CELLRANGE]

[CELLRANGE]

[CELLRANGE] [CELLRANGE]

[CELLRANGE] [CELLRANGE][CELLRANGE]

100

105

110

115

120

125

130

135

140

145

150

C1 Kiswahili C1 English C2 Kiswahili C2 English All

Nth

less

on c

om

ple

ted

Q1 Q2 Q3 Mean

28

Impact of Tusome on Kiswahili Benchmarks

29

70%

45% 43%

19%

17%

21% 19%

15%

12%

32% 33%

54%

1% 3%

4% 12%

Baseline Midline Baseline Midline

Class 1 Class 2

Fluent45+ CWPM

Emergent17-44 CWPM

Beginning1-16 CWPM

Zero Reader0 CWPM

Reading Performance Categories - Kiswahili

Tusome Effect Sizes (standard deviations)

30

SubtaskEnglish Kiswahili

Class 1 Class 2 Class 1 Class 2

Phoneme segmentation 1.07 2.57 -- --

Letter sound knowledge 0.71 1.63 0.75 1.32

Syllable fluency -- -- 0.66 0.80

Invented/non-word decoding 0.52 0.68 0.45 0.50

Vocabulary 0.48 0.41 -- --

Passage reading (A) 0.67 0.72 0.75 0.71

Reading comprehension (A) 0.40 0.49 0.62 0.69

Passage reading (B) 0.73 0.86 -- --

Reading comprehension (B) 0.75 0.94 -- --

Listening comprehension -- -- 0.52 0.52

Average .67 1.04 .63 .76