Renata Lemos: Does Management Matter for Schools? (May 2016)

Why management in schools? How to measure management? Management matters in schools Drivers of management Conclusion & Next Steps

Does Management Matter for Schools?

Renata Lemos

World Bank and Centre for Economic Performance (LSE)

joint with Nick Bloom, Daniela Scur, Raffaella Sadun, and John VanReenen

SeminarComunidad de Madrid

25 May 2016

1 / 33


Long debate over the importance of management forgrowth and development

Francis Walker wrote a papercalled in 1887 in theQuarterly Journal ofEconomics called “TheSources of Business Profits”.

Walker argued thatmanagement was the keydriver of differences inperformance.

2 / 33


Yet many people claim management is “hot air”

“No potential driving factor of productivity has seen a higher ratio ofspeculation to empirical study”

- Syverson (2011)

3 / 33


Why look at management practices in schools?

• Major disparities in the quality of education within and betweencountries

• Hundreds of studies investigating the impact of varying specificinputs into education show substantial heterogeneity across contextsin what works, what doesn’t, and why (Glewwe and Muralidharan2015)

• Mixed views on how to address this

• More recent research points to importance of managerial choices• Charter schools studies (Abdulkadiroglu et al. 2011; Angrist et al,

2013)• Role of basic teaching & managerial practices (Rockoff et al. 2011,

Dobbie and Fryer, 2011, 2013; Fryer, 2014)

4 / 33


Management practices data in schools not systematicallycollected until recently...

[E]fforts to manipulate key educational inputs have been hampered by aninability to identify school inputs that predict student achievement(Hanushek 1997). This is due, at least in part, to a paucity of detaileddata on the strategies and operations of schools...Measures ofteacher development, data driven instruction, school culture, and studentexpectations have never been collected systematically, despite decades ofqualitative research suggesting their importance.

- Dobbie and Fryer (2014)

5 / 33


Summary of findings

• First descriptive evidence on international differences in managementin schools

• Patterns remarkably similar to other industries• Management positively associated with school performance• Wide heterogeneity in school management, even within same

institutional context

• Autonomous government schools outperform in terms ofmanagement relative to regular government schools or privateschools

• Differences in management across these school types are largelyaccounted for by differences in school directors’ accountability andleadership skills

6 / 33


But how to measure management?

• World Management Survey (WMS)

telephone interviews across 35 countries

• Developing WMS

face-to-face interview in the public sector for developing countries

• Management and Organizational Practices Census Survey(MOPS)

self-reported questionnaire in the US

face-to-face interviews in Pakistan and Mexico

• Management, Organization and Innovation Survey (MOI)

face-to-face interview in Eastern Europe

Notes: Bloom, Lemos, Sadun, Scur, and Van Reenen (2016).

7 / 33


But how to measure management?

• World Management Survey (WMS)

telephone interviews across 35 countries

• Developing WMS

face-to-face interview in the public sector for developing countries

• Management and Organizational Practices Census Survey(MOPS)

self-reported questionnaire in the US

face-to-face interviews in Pakistan and Mexico

• Management, Organization and Innovation Survey (MOI)

face-to-face interview in Eastern Europe

Notes: Bloom, Lemos, Sadun, Scur, and Van Reenen (2016).

8 / 33


Our data

• Large international dataset detailing management practices ineducation following the World Management Survey methodology

• Brazil, Canada, Germany, India, Italy, Sweden, UK and US, collectedfrom 2009 to 2013

• 1,851 school directors (schools must teach 15 yr olds & have atleast 50 students; e.g. high schools in US)

• Random sample (response rate of 42%)

Sampling Frame Response Rates

9 / 33


Our methodology

• Developing management questions, WMS style• 20-practice scorecard: use of data, monitoring, targets, incentives• 16 practices similar to manufacturing, hospitals and retail surveys• Scoring Guidelines: 1 to 5

• Getting schools to participate in a 1-hour interview• Conversation is confidential!• Pupil outcomes not asked during the interview• Endorsement letters, e.g. UK Department for Education• Run by 40 MBA-types (loud, assertive & experienced)

• Obtaining unbiased responses• Open-ended questions• Double-blind technique• Double-scoring exercise (69% of interviews)

Sampling Frame Response Rates

10 / 33


The scoring guide

• The WMS gets at what is actually happening at the school, ratherthan how the school director feels or what the school directors thinks

• Across the 20 management topics, we give each school a score of 1to 5, where roughly speaking:

• 1 refers to practically no structured management practicesimplemented

• 2 refers to some informal practices implemented, but mostly areactive approach to management

• 3 refers to a formal process implemented, but with some weakness• 4 refers to a good, formal process in place though not yet

often/consistent enough, but mostly a proactive approach tomanagement

• 5 refers to best practice in the sector

11 / 33


Example Q: Standardization of Instructional Processes

1 How structured or standardised are the instructional planningprocesses across the school?

2 What tools and resources are provided to teachers (e.g.standards-based lesson plans and textbooks)to ensure consistentlevel of quality in delivery across classrooms?

3 What are the expectations for the use of these resources andtechniques?

Score• 1 ⇒ No clear or institutionalized instructional planning processes or

protocols exist; little verification or follow-up is done to ensureconsistency across classrooms.

• 3 ⇒ School has defined instructional planning processes or protocolsto support instructional strategies and materials and incorporatesome flexibility to meet students needs; monitoring is only adequate.

• 5 ⇒ School has implemented a clearly defined instructional planningprocess designed to align instructional strategies and materials withlearning expectations and incorporate flexibility to meet studentneeds.

12 / 33


Example Q: Performance Review

1 How often do you review school performance with teachers andstaff?

2 Could you walk me through the steps you go through in a processreview?

3 Who is involved in these meetings? Who gets to see the results ofthis review?

4 What sort of follow-up plan would you leave these meetings with?

Score

• 1 ⇒ Performance is reviewed infrequently or in an un-meaningfulway e.g. only success or failure is noted.

• 3 ⇒ Performance is reviewed periodically with both successes andfailures identified. Results are communicated to senior staff. Noclear follow up plan is adopted.

• 5 ⇒ Performance is continually reviewed, based on the indicatorstracked. All aspects are followed up to ensure continuousimprovement. Results are communicated to all staff.

13 / 33


Example Q: Data Driven Planning and Student Transitions

1 Is data used to inform planning and strategies?

2 If so, how is it used especially in regards to student transitionsthrough grades/levels?

3 What drove the move towards more data-driven planning-tracking?

Score

• 1 ⇒ School may be aware of critical transitions for students, butlittle or no effort is made to match support services to students;data is often unavailable or difficult to use

• 3 ⇒ School may understand the critical transition points forstudents, although these are not identified in a consistent manner;some data is available, although not necessarily in an integrated oreasy to use manner

• 5 ⇒ Student transitions are managed in an integrated and proactivemanner, supported by formative assessments tightly linked tolearning expectations; data is widely available and easy to use

14 / 33


What we hear from school directors on the phones

Standardized processes?

Interviewer: How standardized are your instructional planning processes?

School director: Very standardized! For example, I tell all my WorldHistory teachers that they must kill Napoleon before Christmas!

15 / 33


What we observe during our school visits

Some compilation (but not so much the use) of data from student reportcards

We visited the schools in AP, India in mid-2013.

16 / 33



One private school had a clean classroom and a posted institutional plan

We visited the schools in AP, India in mid-2013.17 / 33



One government school had an teacher evaluation plan posted on the wall

We visited the schools in AP, India in mid-2013.18 / 33


The adoption of managerial processes is fairly limited, butthere are significant differences across countries

1.7

2.0

2.1

2.5

2.8

2.8

2.8

2.9

1.5 1.7 1.9 2.1 2.3 2.5 2.7 2.9 3.1

India

Brazil

Italy

Germany

US

Canada

Sweden

UK

Figure: Cross country rankings of mean school scores1

1Notes: Data from 1,851 obs. (513 BR, 146 CA, 140 GE, 318 IN, 284 IT, 88 SW, 92 UK, 270US schools). The rankings are similar when we include controls for school characteristics (schoolsize, pupil/teacher ratio, ownership, and presence of academic criteria to select students), schooldirector characteristics (tenure and seniority) and noise controls. 19 / 33


The index distribution shows inequality in schoolmanagement within countries

0.5

11.

52

0.5

11.

52

0.5

11.

52

1 2 3 4

1 2 3 4 1 2 3 4

1 UK 2 Sweden 3 Canada

4 US 5 Germany 6 Italy

7 Brazil 8 India

Graphs by Country

Figure: School management distribution across countries2

2Notes: Data from 1,851 obs. (513 BR, 146 CA, 140 GE, 318 IN, 284 IT, 88 SW, 92 UK, 270US schools)

20 / 33


Substantial within country variation in people management

0.5

11.

50

.51

1.5

0.5

11.

5

1 2 3 4

1 2 3 4 1 2 3 4

1 UK 2 Sweden 3 US

4 Canada 5 Germany 6 Italy

7 Brazil 8 India

Operations Talent

x

Graphs by Country

Figure: School people and ops management distribution across countries3

3Notes: Data from 1,851 obs. (513 BR, 146 CA, 140 GE, 318 IN, 284 IT, 88 SW, 92 UK, 270US schools)

21 / 33


Schools have lower management scores than hospitals(which have worse scores than private manufacturing)

Figure: Management distributions across sectors4

4Notes: The management index is constructed from the 16 questions that overlapin all three sectors. Smoothed kernel density shown for each sector. US Sample,similar in other countries. 22 / 33


We identify different levels of autonomy across thedifferent school types within countries

Table: School type classification 5

5Autonomous government schools are Escolas de Referencia, Brazil; Separate Schools, Canada;Private Ersatzschulen, Germany; Private Aided Schools, India; Friskolor, Sweden; Academies,Foundations and Voluntary Aided Schools, UK; Charter and Magnet Schools, US. There are noautonomous government schools in Italy.

23 / 33


Is our management index related to meaningful educationaloutcomes?

• Not able to establish causality in this research, but useful validationexercise of our data

• With this caveat, run the following regression:

Yic = βmMic + βAUTGOVAUTGOVic + βPRIPRIVATEic + γ′Xic + uic (1)

where,Yic = z-scored measure of school level educational outcomesMic = our survey measure of management qualityAUTGOVic = dummy denoting autonomous government schoolsPRIVATEic = dummy denoting private schoolsXic = measures of school resources/inputs and pupil composition(country-specific)and controls for survey measurement error

24 / 33


Management quality is positively and significantlycorrelated with pupil outcomes (1)

−0.288N=201

−0.052N=200

−0.010N=201

0.090N=200

0.296N=200

−.4

−.2

0.2

.4S

choo

l−le

vel p

upil

outc

omes

(z−

scor

e)

Bottom quintile 2nd quintile 3rd quintile 4th quintile Top quintile

Figure: School management vs school Performance 6

6Notes: Controls added. Performance measures for 1,002 obs: 472 BR, 77 CA, 152 IN, 82 SW,86 UK, 133 US schools

25 / 33


Management quality is positively and significantlycorrelated with pupil outcomes (2)

Table: School management vs school performance 7

7Notes. Significance at the 1% level denoted by *** and ** for 5% and *10% levels. OLSestimates with robust standard errors in parentheses under coefficients. Management is z-score ofthe averaged of the zscored 20 individual questions. All regressions have country dummies.General controls: regional dummies, school curriculum (academic versus vocational) and noise (jobpost and tenure of interviewee; interviewer dummies, day of week; time of day and interviewduration and reliability measure). Pupil controls are country-specific.

26 / 33


Pupil outcomes & decompositions of management scores

7Notes. Significance at the 1% level denoted by *** and ** for 5% and *10% levels. OLSestimates with robust standard errors in parentheses under coefficients. Management is z-score ofthe averaged of the zscored 20 individual questions. All regressions have country dummies.General controls: regional dummies, school curriculum (academic versus vocational) and noise (jobpost and tenure of interviewee; interviewer dummies, day of week; time of day and interviewduration and reliability measure).

27 / 33


What accounts for differences between schools?

• We focus in particular on governance, ownership, accountability andleadership issues.

• We use a simple regression model of the form:

Mic = αAUTGOVAUTGOVic + αPRIPRIVATEic + γXXic + uic (2)

where,Yic = z-scored measure of school level educational outcomesMic = our survey measure of management qualityAUTGOVic = dummy denoting autonomous government schoolsPRIVATEic = dummy denoting private schoolsXic = measures of school resources/inputs and pupil composition(country-specific)and controls for survey measurement error

28 / 33


We looked at a range of school inputs and characteristics...

• no of students, student/teacher ratio

• students selected on academic merit

• regular (non-vocational) curriculum

• population density

• competition

• director’s tenure, gender, STEM background

• director’s personnel, budgetary, academic content autonomy

• And found that governance/accountability and leadership skillsof school directors matter the most!

29 / 33


Conclusion

• First descriptive evidence on international differences in managementin schools

• Patterns remarkably similar to other industries

• Autonomous government schools perform the best in terms ofmanagement

• Their better performance is not linked with autonomy per se butwith how autonomy is used

• Governance and leadership skills of school directors: two keyfeatures that account for a large fraction of the better managementof government autonomous schools

30 / 33


Next steps

• More management surveys!

• RCTs on management and leadership training of school directorsand pedagogical coordinators

• primary and lower secondary schools in 7 states in Mexico, andpotentially rolling up to a Census Survey (with Rafa de Hoyos,Daniela Scur, and the Mexican Ministry of Education)

• early childhood education centres in Brazil (with Rita Almeida,Leandro Costa, Daniela Scur, and the Ministry of Education)

31 / 33


A quote from one of our lovely school directors

An appraisal system?

Interviewer: How do you evaluate your teachers?

Principal: Well, I try to have a review at the end of each semester...but itis difficult...my teachers are terrified of me. One teacher has missed hisreview twice already. He came back the next day saying he was abductedby aliens...twice...C’mon!

32 / 33


Does Management Matter for Schools?

Renata Lemos

World Bank and Centre for Economic Performance (LSE)

joint with Nick Bloom, Daniela Scur, Raffaella Sadun, and John VanReenen

SeminarComunidad de Madrid

25 May 2016

33 / 33

EXTRAS

The Survey Response Rates by Country

Back

1 / 2

EXTRAS

Sampling Frame & Sources

• Brazil: Instituto Nacional de Estudos e Pesquisas Educacionais (INEP)

• Canada: Scotts Directories (Private company compiling information for all schools inCanada)

• India: District Information System for Education (DISE), Central Board for SecondaryEducation (CBSE), Indian Council of Secondary Education (ICSE)

• Italy: Ministero dell’Istruzione, dell’Universit e della Ricerca

• Sweden: Skolverket (Swedish National Agency for Education)

• Germany: Various state departments

• United States: National Center for Education Statistics

• United Kingdom: Department for Education

Back

2 / 2

Renata Lemos: Does Management Matter for Schools? (May 2016)

Education

Transcript of Renata Lemos: Does Management Matter for Schools? (May 2016)