Think systematically - Serious Social Investing 2013

April 10, 2023

When M&E _is_ the intervention

District management through data

© 2013 Michael & Susan Dell Foundation Company Confidential 2

We have drawn on lessons from the Michael & Susan Dell Foundation’s global experience in performance management in education

2

The Michael & Susan Dell Foundation has invested $160M+ in grants in multiple districts across 9 states in the US for educational performance management.

In addition, the foundation has embarked upon ambitious partnerships with government in India to measurably improve learner results across major metros.


Key success factors for education data systems identified through the experience of the Dell family foundation

3

Easing the data collection burden must be a central driver of developing the new application – it must make people’s lives easier if it is to be successfully adopted

Thinking beyond just accountability, to actual value to the end user, is essential for success

Tools must enable a shift in district role with regard to schools from compliance-focused to support-focused

Stakeholder engagement, including extensive market research with users, is essential to get the application right and to ensure uptake upon roll out

Building an ecosystem of training and other support organizations from the earliest pilot is vital to achieve scale

Proven dashboards, data standards and other protocols from the foundation’s work in the US could be leveraged in South Africa


Why should South Africa engage with education data systems now?

• Wealth of new learner performance data now available. Limited systems to actually enable schools and districts to use it

• District directors manage organizations the size of medium to large businesses with little to no information to guide them

• Without an in built system to monitor performance, the education system has no way to know which interventions work and which don’t

Together with the DBE, in 2012 we conducted a diagnostic study of data at every level of the school system across all 9 provinces

Research across all 9 provinces

Note: Infrastructure descriptions: Poor IT = Less than 2 computers, no phones, no internet // Fair IT = 2 - 5 computers used by staff, maybe internet connected and phone linesGood IT = More than 5 computers, phone lines and internet, printers, copiers etc. Below average, Average and Above average determined by using reported pass rate – with 40-60% being considered average – considers both primary and secondary schools

Western CapeEastern Cape

Northern Cape

Free stateKwazulu Natal

North WestMpumalanga

Limpopo

▪ Over 250 interviews

▪ National-, Province-, district-, school-, and class-level interview

▪ Diverse and representative mix of educational environments: – Mix of urban and rural

schools– Distribution of wealth

quintiles– Spectrum of PC and

internet access– Range of performance

bands

Deep-dive visits (2 weeks)

Light touch visits (1-2 days)

School locations

Gauteng

5


What did we learn?

7

Despite collection and recollection of data, schools, circuits and districts receive little to no insight or feedback on submissions

NationalEMIS

Generic Reports

Public Queries

Provincial EMIS

Province directors

District EMISDistrict director

SMGD

?

National directors

Other data requests

Data flows upwards to satisfy compliance requirements

School

Schools are overburdened by data requests

Duplication of data requests on schools

-68%

“Net new” questions

183

Questions duplicated across set of surveys

144

Questions with answers in SA-SAMS submissions

245

Questions asked by district tools

572

Number of survey questions asked by district per quarter

EXAMPLE

How does this district analyse 500+ questions per quarter per school?

How does this district analyse 500+ questions per quarter per school?

8

Outside of planning, few management routines use data to guide decision making

Learner demographic and ID info

Learner registration and promotion

Learner daily attendance

Learner subjects and timetable

Leaner marks (daily, weekly, quarterly)

Learner grade performance and pass quality

Learner scores for standard provincial assessments

Learner scores for standard national assessments

Learner social needs and discipline record

Educator ID, qualifications and assignment

Educator curriculum coverage

Educator development and training

Educator attendance and leave

Educator teaching and learning results and trends*

School ID and contact information

School infrastructure and facilities

School finances

School LTSM ordering and delivery

School improvement plans

School posts vacancies and appointments

School support services - transport and feeding

School teaching and learning results and trends**

School management meetings

Circuit and district management visits

Province, district, and/or circuit level goals/targets

Metric requested at province level

Common focus goals

Capture data to

measure progress

Input data in

shareable system

Aggregate and

analyse data

Create actionable

outputs

Establish skills and routines

System-wide difficulty in translating collected data into insights and actionsSystem-wide difficulty in translating collected data into insights and actions

9

Our biggest surprise…

Government’s response

10


What might the solution be?

Districts are a critical point of leverage in the school system

12

National DBENational DBE

Provincial DoE (9)Provincial DoE (9)

Districts (86)Districts (86)

Schools (~27 000)Schools (~27 000)

ClassroomsClassrooms

Primary levers for improving learner performance

▪ Strategy, policy and direction▪ Design and delivery of instructional support material▪ Educational system resourcing

▪ Strategy, policy and direction▪ Human resources and hiring▪ District and school resourcing

▪ Instructional and administrative support and tools▪ Performance management▪ Quality assurance and compliance▪ Professional development▪ School and community engagement

▪ Local and operational support▪ Enrolment and progression▪ Delivery of facilities instructional material

▪ Teaching and learning

Performance management

improvements at the district level can put learner performance at the forefront of

thought in classrooms and

provincial offices alike

Performance management

improvements at the district level can put learner performance at the forefront of

thought in classrooms and

provincial offices alike

Districts are also historically the most ignored and atrophied part of the delivery chain across South Africa

Districts are also historically the most ignored and atrophied part of the delivery chain across South Africa

What will it take to establish 86 data-driven school districts?

13

Action / Routines

Script data-driven management routines for districts to support and review schools’ performance

2

Analyse

Outputs / Dashboard

Put data in the hands of people who can act on it by creating data dashboards that:• are visually easy to interpret,• trigger action where it is needed• display analyses for root-cause identification, • designed into management routines, and • are user-tested

3

Input and aggregate

Collection

Streamline and strengthen data collection processes to [1] improve accuracy and [2] reduce duplication & burden

4

Identify and build capabilities and mindsets to implement data-driven action

5

Scale nationally, pioneering first across 3 districts that represent a wide range of district contexts, ‘to meet districts where they are’

6

MetricsAgree on a small-set of student outcome metrics across grades R-12, aligned at province, district, & school level1

Work in close partnership with government

7

We have selected districts of three different types and have begun work with them to map processes and design a dashboard

14

National DBENational DBE

LimpopoLimpopo Free StateFree State GautengGauteng

Waterberg Thabo Mafutsanya Ekudibeng

Consulting and operations team: McKinseyConsulting and operations team: McKinsey

SI and Design: Frog Design, Double Line PartnersSI and Design: Frog Design, Double Line Partners

Training and coaching organizations: New Leaders FoundationTraining and coaching organizations: New Leaders Foundation

“Builder” pioneer

Overstretched district in rural area with large number of schools per Circuit Manager

Limited technology in place

Matric results only real indicator managed

“Architect” pioneer

Similarly rural setting, but with more technology infrastructure and support in place

Practice of using data to manage for results established

“Experimenter” pioneer

Functioning dashboard developed in Excel, tracking a full range of indicators

Management for results change process underway for 2 years

Other 6 provincesOther 6 provinces

“Twin” districts

• Shadows process• Reality checks work

to ensure generalizability

One per province

A district management system can support the transition to data-driven performance through a combination of processes, tools and capabilities

15

Building KPIs and business processes Establishing tools and capabilities

60%60% 40%40%

Business processes and tools will be tailored to meet districts’ schooling and technology realitiesBusiness processes and tools will be tailored to meet districts’ schooling and technology realities

Work has started to create visually appealing dashboards for different district archetypes

16

Examples of exercises from the co-development of dashboards with district officials

18

Participants felt the workshop was an “eye- opener” and it motivated them to improve current practices

22

I feel empowered to change

My eyes are open now…we are able to read the statistics and see how they relate

You can pick up problems through the data

It was not boring, it was like playing and now we are empowered

The pilot would be a good tool for us to use in the district

It seemed like child’s play at first but now we are empowered

” ”

It shows we have been haphazard in the way we have been doing things…this is the start of change

Source: Waterberg design workshop

We have identified a core set of student and school metrics to guide school performance support and management

SOURCE: Team analysis

Indicators Unit of measure Range Archetype

1 Annual National Assessment, National senior Certificate (Matric) 2 Written test, standardised across the region/province, data with/without SBA 3 100% TBD

Attendance

Curriculum

Inputs

Resourcing

Learner outputs

Achievement

Progression

▪ % of days ▪ Learner attendance 0 - 100

▪ % of days ▪ Educator attendance 0 - 100

▪ All

▪ All

▪ Experimenter▪ % of curriculum▪ Option 1: Common Tests2 0 -100

▪ Experimenter, Architect

▪ %3 ▪ LTSM 0-100

▪ %▪ Vacant educator positions

0-100 ▪ All

▪ All▪ Levels, %▪ ANA, NSC1 1-7, 0-100

▪ Experimenter▪ Levels, %▪ Common Tests2 1-7, 0-100

▪ All

▪ All

▪ %

▪ %

▪ Grades 9, 10, 11 pass rates

▪ Grade 12/Grade 8 learners

0-100

0-100

▪ All▪ %▪ Matric pass rate 0-100

▪ Ekudibeng▪ % of curriculum▪ Option 2: CAPS syllabus coverage

0 -100

23

PRELIMINARY

We are developing a baseline which will track core metrics along with operational improvements and school feedback

Education outcomes

▪ Measures of learner achievement and progression at all stages of the basic education system

▪ Measures of enablers that contribute to achievement and progression improvements

▪ Improvements in achievement and progression are the ultimate goal for DBE and all stakeholders

▪ Enablers are the leading indicators for performance in achievement and progression

▪ Long-term

▪ Medium-term

Operational improvements

▪ Measures of the effectiveness of data-driven management processes

▪ Measures of the efficiency of data processes

▪ Operational indicators measure the levers that the project will use to drive improvements in education outcomes

▪ Short-term

Feedback from schools

▪ Measures of the schools’ assessment of the quality, frequency and impact of their interactions with the district

▪ Schools are in the best position to assess the shift form compliance to support and give districts feedback on their service to schools

▪ Medium-term

Description Rationale Time to impact

SOURCE: Team analysis

National rollout

National rollout

Preliminary timeline for pioneering districts

25

PRELIMINARY

DBE leads roll

out together

with provinces

Prepare for implementation

▪ KPIs

▪ Map of data collection, storage, and use

▪ Design process changes

▪ Develop dashboard prototype

▪ Assess baseline of district effectiveness

▪ Plan for implementation & roll-out

3 months3 months

Begin implementation

▪ Implement change process within pioneer districts

▪ Develop actual dashboard system

▪ Forge partnerships to expand funding base

6 months6 months 12 months12 months TBC…TBC…

▪ Work with district, circuit and schools to embed tools and practices

▪ Finalize reporting structures to province and national

▪ Package materials and prepare for next district rollouts

All

dist

rict t

ypes

All

dist

rict t

ypes

▪ Deliver online dashboard, linked to existing data systems

▪ Deliver training

“Exp

erim

ente

rs”

“Exp

erim

ente

rs”

▪ Create local printed outputs and traing

▪ Shift output and analysis capabilities to province“B

uild

ers”

“Bui

lder

s”

▪ Deliver offline dashboard, drawing from existing data systems

▪ Conduct dashboard and report training

“Arc

hite

cts”

“Arc

hite

cts”

All

dist

rict t

ypes

All

dist

rict t

ypes

= Current phase

Beyond 3 months the timeline is subject to revision as we proceed


In Closing, Some Broader Reflections

• Consider the power of M&E as the intervention itself – giving government and NGOs the tools and training to manage their own organizations for success

• Appreciate the context into which our data requests as donors fall‒ How can we mirror the existing reporting cycle at the

grantee? How can we align with their other reporting requirements?

‒ Can we use existing data? Can we leave behind an M&E system that sustainably benefits the grantee / government entity?

Thank you!


Comments to reference from Day One

“Most M&E within the government is managed from a compliance perspective. People report because they have to, and do not use the M&E information to manage their departments.”

“Accountability is to late. The most important thing is to build a management culture of continuous improvement.”

Dr. Ian Goldman, The Presidency

“One of our most valuable contributions in our jointly setting agenda with government partners is being able to help them set more realistic targets for improvement, that are based on a clear understanding of where you are today and what level of change is possible in the time period.”

Gail Campbell, Zenex Foundation

Findings suggest that stakeholders face common challenges across all phases of a data-driven decision-making cycle

29

Ass

essm

ent

Ass

essm

ent

An

alys

isA

ctio

n

Create shared focus on measurable goals

Input data into a sharable system

Capture data to measure progress

against goals

Aggregate and analyse data

Create actionable outputs for various

users

Establish skills and routines to review data

and guide action

Schools are overburdened by reporting requirements, which exist in duplicate and triplicate because districts, provinces and national offices do not share data

Outside of planning, few management routines use data to guide decision-making, instead most rely on touch and feel and ad hoc efforts to manage crises

Despite its promise, SA-SAMS is not being used as a school management tool, and is primarily used by administrative clerks as an “electronic accountability tool”

Stated goals and incentives do not always line up, resulting in managers prioritising the items that lead to funding or publicity above all else

Data driven decision cycle Research themes Example analyses*

* Detailed view of analyses can be found in appendix and supporting documents

There is no single source of accurate data for circuits or districts, as each function sources its own data and EMIS remains outside of core conversations

District, circuit and school personnel do not receive feedback on submitted data, as most data is collected and passed upwards in compliance, with limited desire to inform ground-level actors of relevant data insights

Our current work in implementation preparation is guided by six key questions

30

How can data best be optimised and used to support schools to improve performance?

What are the data-driven routines that districts should put in place for supporting schools and managing their performance on a regular (termly) basis to drive lasting improvements in education outcomes?

How can data systems (inputs, processes and access) be optimised to ensure that accurate and unique inputs result in relevant information that supports data driven support and performance management?

How can school data be best displayed to result in the most effective action by school officials, politicians and other decision makers to improve R-12 education across all districts?

How will we build the capabilities and mindsets needed to drive data-driven school support?

How do we plan for a successful rollout to all districts in the country?

2

3

4

5

6

What are the core guiding metrics for data-driven school performance management, and what is the current baseline of performance and practices?

1

Think systematically - Serious Social Investing 2013

Education

Transcript of Think systematically - Serious Social Investing 2013