Governance: An example and what Bank evaluations can addJ ISHNU DAS, DECRG

INSIDE GOVERNMENT WORKSHOP

WASHINGTON D.C. , FEBRUARY 2016

Prelude: An Indian Puzzle

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

Medium Performing States

Public MBBS

Public AYUSH

Private MBBS

Private AYUSH

Private none

Low Performing States

80 percent of India’s primary health care providers do not have any formal medical training

Prelude: An Indian Puzzle

0

10

20

30

40

50

60

70

80

90

100

MBBS

AYUSH

Other

Although within each state, the providers without formal medical education know less, there are many states where these providers know more than fully qualified medical doctors in other states

Prelude

You have to wonder: Why are so many medical doctors in India so poorly trained?

This presentation

Use this motivating example to ◦ understand proximate (training is poor) versus institutional (institutions are structured so that training is

poor) constraints

◦ Argue that World Bank can add significantly to analytical narratives that deepen our understanding of the link between institutions and proximate outcomes

Briefly try to understand whether we are positioned to do this◦ Return to proximate versus institutional constraints in WB evaluations: The governance of governance

evaluations

Caveat: Much of this is based on my own experience and from health and education; No claims to external validity

A starting point

Times of India: Jan 20th 2016

Black money quota: 25,000MBBS PG seats at Rs.12,000 crores◦ Lots of people are paying bribes to go to medical colleges

◦ “Many of these colleges are running with little or no facilities, no patients and fake facilities”

CNBC reports on medical colleges hiring patients to lie in beds to show enough patients to provide students with clinical experience to government inspectors in Muzaffarnagar

◦ “I am not very keen to reply” said Dr. Anil Agarwal, the school’s principal

The Vyapam scam

The Vyapam scam is an admission & recruitment scam involving politicians, senior officials and businessmen in the Indian state of Madhya Pradesh. Madhya Pradesh Professional Examination Board (MPPEB), popularly known by its Hindi acronym "Vyapam" (Vyavsayik Pariksha Mandal), is a self-financed and autonomous body incorporated by the State government responsible for conducting several entrance tests in the state. These entrance exams are held for recruitment in government jobs and admissions in educational institutes of the state. The scam involved a collusion of undeserving candidates, who bribed politicians and MPPEB officials through middlemen, to get high ranks in these entrance tests

A number of people connected to the scam and its investigation, died during the course of investigation. The opposition parties and activists alleged that several of these deaths happened under suspicious circumstances. In 2015, the Special Task Force (STF) submitted a list to High Court, naming 23 people who are believed to have died "unnatural death". According to the STF, most of these deaths took place before it took over the investigation in July 2013. Some media reports claimed that 40+ people associated with the scam had died under mysterious circumstance

A puzzleWhy are people willing to pay large sums of money to attend what appear to be “degree mills”?

Returns to a degree must be worth the money either in the public or the private sector

Returns to performance in the public and private sector

0.0

1.0

2.0

3.0

4

freq

ue

ncy

9.5

10

10.5

11

wag

es

0 20 40 60Checklist Items Completed

Fitted values Log of salary

kdensity percent_recqe

Wages and Performance in the Public Sector

0.0

05

.01

.01

5.0

2

freq

ue

ncy

02

04

06

08

01

00

fee

s

0 20 40 60 80Checklist Items Completed

lowess price percent_recqe Cost of interaction

kdensity percent_recqe

Fees and Performance in the Private Sector

Two further notes1. Public sector pay is much higher than private revenue for the vast majority of doctors2. Except potentially for the top 1-5%

Training equilibrium: PossibilitiesThe return to a degree exceeds the bribe even in the private sector. The schools add something. Possible: Returns to just a degree are around Rs.30 per patient, which can add up

The degree provides eligibility to the doctors to join the public sector◦ Supreme Court has often taken stance of equal treatment for equal degrees

◦ This is a real possibility; interviews suggest that many such providers are aiming for a public sector job, where the quality oftraining seems not to matter

Equilibrium◦ Some colleges provide minimal training: Degree mills with graduates going to public sector

◦ Some colleges provide excellent training: Graduates go to high paying private sector or emigrate

Proximate versus InstitutionalWorld Bank Project: Improve the quality of medical training

Step 1: No or very limited background research on understanding the problem

Step 2: Improves medical training through “capacity building” and “peer group monitoring using quality collaborative with change management cutting edge techniques”

Step 3: Design: Evaluation team randomizes 100 medical colleges into treatment and control group. Checks power and balance.

Step 4: Surveys: Baseline, midline and endline, but problems due to implementation delays and deviations from protocol

Step 5: Paper: “The causal impact of capacity building and peer group monitoring on medical education: A structural estimation”

Step 6: Return to status quo within X years

What did we really learn?

InstitutionalGovernance problem composed of two (maybe three) parts

No link between pay and quality of training in public sector◦ Relevant evaluation: What happens if salaries reflect actual knowledge in the public sector?

Poor regulation of medical colleges unable to address problems from first part; degrees granted by universities

◦ Feasible Relevant evaluation: Contracted out testing of medical students to a third-party auditor

◦ Note the parallel to the institutional structure of schooling, law or medicine in the U.S. where provision is decentralized, but testing is centralized at the state/national level

◦ If you are not going to rely on returns, the auditors cannot be the providers

Perhaps restricted entry into setting up medical colleges◦ Relevant retrospective work: Impact of entry of medical colleges on doctors wages

InstitutionalEach of these evaluations require substantial legwork and consensus to implement

◦ Pay issues difficult to solve: Supreme Court favors payment on observables (degrees) over observable process, on the grounds that the latter can lead to favoritism and corruption (admittedly a potential issue)

◦ Regulation difficult to solve: Head of the Medical Council of India (the regulators) has been under investigation of and on for taking bribes from colleges

InstitutionalShould we then give up? Answer is an emphatic no

I would argue for fundamental knowledge that helps◦ A. Build analytical narrative to

◦ understand the link between institutions and their logic and measured outcomes. These contribute critically to democratic discussion and debate, building constituencies for institutional change

◦ attack portions of the problem not currently studied: One example is understanding performance variation within a given institutional arrangement, which is substantial and often much larger than in OECD countries

◦ B. Demonstrate in “gold standard” studies the impact of simulated institutional change. These have no external validity for what would happen if the government were to do it, but that’s not the point: The provide a model of “if you can do this, you would get that”

◦ C. Evaluate proximate solutions, especially those that are commonly used: These are premised on the null hypothesis of a zero effect, taken seriously

The need for an Evaluation 3.0 modelWe are caught between the seduction of the proximate and the power of the institutions

But there are key evaluations we should be doing that go beyond the proximate

World Bank has been conducting evaluations for over a decade now, with substantial learning about evaluations themselves

Model 1.0 was getting things started

Model 2.0 was understanding what an evaluation entails

Model 3.0 should◦ Discuss what World Bank evaluations should focus on

◦ Discuss what we need to get there

Evaluation 3.0: What should the WB focus on?World bank focus on “stuff that’s not getting done elsewhere”

Analytical narratives satisfy this for several reasons◦ Imply “thick” descriptions of situations and definitions of problems in countries that may be less

prominent

◦ Requires substantial investment and difficult to publish well—at least at first◦ But potentially significant dividends in terms of crowding-in research and changing how aid functions: WDR 2004 on Service

Delivery

The work “that is not getting done elsewhere” does not necessarily include evaluations of Bank programs, that can be financed but not provided

◦ This allows for firewalls between implementers and evaluators

Evaluation 3.0: What do we need to get there?Incorporate the learning from 1.0 and 2.0 to model how evaluations work in practice and the logic of why they work the way they do

Evaluation 3.0: The Governance of Governance evaluations

Co

sts and

Ben

efits

Years

Implementation PublicationBaseline/take-up

Payoff to researcher

Payoff to operational staff (not on same scale)

Ideal Cost to researcher

Ideal Cost to implementors

Proximate problems versus institutional issuesProximate problems: Evaluations are delayed, implementers are non-responsive, evaluation team has to put in substantial effort into implementation, many evaluations do not come to fruition

◦ Notable exceptions to this picture

Institutional logic: ◦ Under existing arrangements, operational incentives are to lend and implementation (especially under the

rigors of an RCT) is less of a concern◦ For example: Have a conversation with a manager where you say that realistically, the evaluation will take up 50% of the Task Team Leaders

time for the year if done well

◦ For example: Supervision budgets are much smaller than preparation budgets

◦ One way to complete an evaluation is to force the researcher to bear the implementation risk, and raise money for supervision, through implementation monitoring

◦ This is done by funding for the entire evaluation, with researcher bearing the cost of failure in implementation: Use economies of scope

◦ Researchers put in substantial work at the beginning and see very little payoff till publication, so they are stuck

Proximate problems versus institutional issues

Equilibrium

Researchers know that implementation risk is very high.

Option 1: Stick it out and run the implementation. Worth it if you are sure that this is gold.

Option 2: Diversify the portfolio. Spread funding risk among multiple projects, invest less in project until analysis and writing phase. Usually this is the domain of senior co-authors, but now many staff have evaluation portfolios of 10-12 projects

Low investment in learning from the evaluation unless the implementation bears fruit (leading to the “why didn’t I collect that information in the baseline to begin with” problem); many evaluations delayed or not completed.

Ultimately, we are unable to advance the analytical narrative

Evaluation 3.0: Governance vs. Governance

We need to talk: Seven things that need to be worked out

1. Where should we act as market makers, not providers—interesting and difficult problem◦ Example: Letters to get people to pay more taxes. Unclear why this cannot be written as an outcome contingent contract, with Bank helping to structure contract

2. Where evaluations did not work, specify why◦ Example: In review of bank documents in South Asia, the word “data” did not enter the CMUs comments even once across multiple projects

3. Where they did work, specify why and where in hierarchy support was critical (or how hierarchy was bypassed)

4. Evaluations of bank programs, specify who bears the implementation risk and how the evaluation is specified if the implementation does not go through/is restructured/is substantially delayed

◦ Example: “After the 5 year evaluation timeline, the impact is 0 because the implementation had not started. We have spent $600,000 on two baselines”

5. Specify ex ante evaluation contracts clearly: For instance, time requirements from WB researchers and staff◦ Example: This is often deflated because we want to show many evaluations that are running concurrently

6. Make sure that we are doing evaluations where we do not have country buy-in◦ Example: Most work on health care met with significant resistance because it did not fit with existing lending structure. If the status quo is not resisting, you are

not changing the status quo

7. Ensure clear property rights◦ Example: We are often told that “ensure that the client owns your idea and the evaluation and the greatest success is when they present it as their own”. Fine, to

a point. At what point does it stop being fine and become theft of intellectual property?

SummaryExciting times in governance with key new learning on

◦ Links between Institutions (big I) and development

◦ Ability to work within existing institutions to diffuse new ideas

◦ Vastly improved measurements in health and education of what services are delivered at the frontline

◦ Ability to leverage new technologies to shift existing equilibrium

Next steps may be deepening linkages between institutions (small i) and performance and the variation in performance within institutional arrangements

◦ But this may require important conversations on the governance of evaluations

Build up knowledge of how evaluations are functioning and improvements◦ This is already happening to some extent: Proposed work on measuring governance

◦ But also need to model the institutional arrangements and examine how evaluations should be structured and completed

Additional slides just in case

Evaluation 3.0: An example of analytical narratives currently understudiedWhy is there These variations are substantial and larger than in OECD contexts

◦ Example: Teacher value-added effects twice as strong in Pakistan relative to U.S. (Bau and Das)

◦ Example: Public schools are among the best, but with a long tail of poor performers

Typically modelled as a non-malleable fixed-effect◦ Policy at selection margin, assuming that fixed-effect becomes observable over time

◦ But we don’t really understand the nature of the fixed-effect, whether it is modified by hierarchical arrangements, the effect of managers, elasticity of the fixed-effect to different types of hiring arrangements etc.

Example: Variation within institutional arrangements

One idea

Director-Supervisor-Agent model

Effort of agent (motivation, morale) depends on strength of links between supervisor and director

If effort requires directorial approval and supervisor is out in the cold, not worth it

See for instance, Paul Grout’s work on collapse of morale in organizations

We really care whether our director has a good relationship with the VP and the VP with the president

Anecdotally: Successful midwives are those who can get the doctor in if the labor is problematic