1

Musings on Governance Indicators,

and some pathways forward…

Daniel Kaufmann

The World Bank Institute

Presentation at Seminar on The Empirics of Governance, May 1-2, 2008, World

Bank, Washington DC

2

Overview

• Preamble: Many extremely valuable efforts…

• Measuring governance/institutional quality– some conceptual / definitional musings– what do we measure? rules vs. outcomes– whose views? experts vs. survey respondents– measurement error– disaggregation, ‘actionability’, & aggregation

• Constructively Moving forward…

3

Do we know what governance is?• Ok: No Agreed standard or Unified Theory (AUT) of governance

• Yet: Absence of AUT is also the case with most any field in social science/economics fields… [& physics..?]

• And: Absence of AUT in many fields hasn’t deter measurement

• In governance (as in other fields), in spite of no AUT: there is a scholarly body of theorizing on governance & institutions

• Further: various definitions have been put forth over the years, mostly they are variants of each other – but we opt against either anorexic or tautologically ‘fat’ definitions – ‘broad’ is ok

• Distinguish between pre-requisites for measuring governance, vs. hypotheses on whether, what, and extent of governance mattering for development outcomes (though iterative process)

• Key question: do conceptual or definitional differences make a 1st or 2nd order difference to the measured indicators (or are other factors more important)?

4

Definitional Issues, in more detail (1)

• Makes sense to avoid overly fat definitions

– risks making links from governance to development tautological (think about North quote)

– risks lack of focus in what we measure

• e.g. Ibrahim Index of African Governance averages together things like corruption, inflation, and per capita GDP

• Also makes sense to avoid overly narrow definitions

– constitutions institutions!

– de facto rules matter more than their de jure counterparts

• Don’t overstate degree of definitional disagreement, most definitions encompass capable states, operating under rule of law, and accountable to citizens

5

What do we mean by “Governance”, Cont’d (2)?

• World Bank (1992): "Governance is the manner in which power is exercised in the management of a country's economic and social resources for development“

• World Bank (2007) definition: "...the manner in which public officials and institutions acquire and exercise the authority to shape public policy and provide public goods and services"

• WGI Definition (1999): "...the traditions and institutions by which authority in a country is exercised. This includes the process by which governments are selected, monitored and replaced; the capacity of the government to effectively formulate and implement sound policies; and the respect of citizens and the state for the institutions that govern economic and social interactions among them."

6

A Fundamental Challenge in Measuring Governance

• The ‘true’ level of governance is inherently unobservable (or, at least, in some dimensions, extremely hard to measure)

• Unavoidably: any measure of governance and investment climate are proxies of the ‘true’ level

• Cats, mousetraps and bathwater filters: the above means that we need to take very seriously: i) triangulation; ii) margins of error; iii) moving from objective-subjective distinction to more meaningful ones…

• Further, different objectives means different tools are more appropriate

7

A Taxonomy of Governance Indicators: moving away from the subjective-objective divide to rules vs outcomes-based

Whose Opinion? About What?

Broad Specific Broad SpecificExperts

Lawyers DBCommercial Risk Rating Agencies DRI, EIU, PRSNon-Governmental Organizations GII HER, RSF, CIR, FRH GII, OBIGovernments and Multilaterals CPIA PEFAAcademics DPI, PIV DPI, PIV

Survey RespondentsFirms ICA, GCS, WCYIndividuals AFR, LBO, GWP

Aggregate Indicators Combining Respondents TI, WGI, MOI

Rules Outcomes

8

Legend for TaxonomyCountries

Code Name Covered Frequency Link

AFR Afrobarometer 18 Every 3 yearsCIR Cingranelli-Richards Human Rights Dataset192 Annual www.humanrightsdata.comCPIA Country Policy and Institutional Assessment136 Annual www.worldbank.orgDB Doing Business 175 Annual www.doingbusiness.orgDPI Database of Political Institutions 178 Annual http://econ.worldbank.orgDRI Global Insight DRI 117 Quarterly www.globalinsight.comEIU Economist Intelligence Unit 120 Quarterly www.eiu.comFRH Freedom House 192 Annual www.freedomhouse.orgGCS Global Competitiveness Survey 117 Annual www.weforum.orgGII Global Integrity Index 41 Every 3 years www.globalintegrity.orgGWP Gallup World Poll 131 Annual www.gallupworldpoll.comHER Heritage Foundation 161 Annual www.heritage.orgICA Investment Climate Surveys 94 Irregular www.investmentclimate.orgLBO Latinobarometro 17 Annual www.latinobarometro.orgMOI Ibrahim Index of African Governance 48 Every 3 years www.moibrahimfoundation.orgOBI Open Budget Index 59 Annual www.openbudgetindex.orgPEFA Public Expenditure and Fiscal Accountability42 Irregular www.pefa.orgPIV Polity IV 161 Annual www.cidcm.umd.edu/polity/PRS Political Risk Services 140 Monthly www.prsgroup.comRSF Reporters Without Borders 165 Annual www.rsf.orgWCY World Competitiveness Yearbook 47 Annual www.imd.ch

9

What To Measure: Rules or Outcomes?

• Rules-based de jure measures capture formal rules, institutions, and processes

– presidential vs. parliamentary system

– first-past-the-post vs. proportional representation

– does a statutorily-independent election monitoring agency or anticorruption agency exist?

• Outcome-based de facto measures capture how these rules are implemented and their effects in practice

– is election monitor politicized?

– do firms perceive corruption is a problem?

– do teachers show up at school?

No “bright line” separating the two, think of as a continuum

10

Rules-Based Indicators

• Main virtue is clarity: does a rule exist? has legislation been passed?– obvious appeal to gov’ts / donors who want to monitor

progress on governance– reflected in language of “actionable” indicators

• Three main drawbacks– easy to overstate clarity: measurement of all but most

basic rules requires careful judgment – virtually no truly “objective” indicators exist

– links from (highly detailed) rules to outcomes not always clear

• which “actionable” indicators are “actionworthy”?• risk of “reform illusion” or “teaching to the test”

– obvious potential for gaps between de jure rules and de facto outcomes

11

Outcome-Based Indicators

• Majority of governance indicators focus on measuring de facto outcomes (governance outcomes, that is…)

– budget outcomes relative to budget plans? (PEFA, OBI)

– fraction of contract value paid as bribe? (GCS)

– is corruption in government widespread? (GWP)

• Main virtue is that these measure relevant governance outcomes as seen by stakeholders, including informality

• Drawbacks:

– (broad) outcome measures hard to link back to specific policy interventions (mirror image of previous problem)

– (poorly-designed) questions can be vague and open to interpretation

• avoid generic questions, focus on experiential ones

• provide benchmarks for respondents (e.g. vignettes)

12

Subjective and Objective Measures of Ease of Business Entry: OECD/NIC Sample

2

3

4

5

0 40 80 120Number of Days to start a Business (DB)

Dif

ficu

lty

of S

tart

ing

a B

usi

nes

s (E

OS

)

Low

High

r = 0.51

13

Subjective and Objective Measures of Ease of Business Entry: Developing Country Sample

1

3

5

7

0 40 80 120 160

Number of Days to start a Business (DB)

Dif

ficu

lty

of S

tart

ing

a B

usin

ess

(EO

S)

Low

High

r = 0.24

17

Are Elections Free and Fair? Global Integrityvs. Global Corruption Barometer Survey

ZAFYUGUSA

SEN

RUS

ROM

PHLPAK

NICNGA

MEXKEN

ISR

IND IDN

GTM

GHA

ETH

BGR

ARG

y = 23.09x + 55.30

R2 = 0.19

y = -3.14x + 85.69

R2 = 0.00

0

20

40

60

80

100

120

0 0.2 0.4 0.6 0.8 1

Voice of the People Household Survey(Are Elections Free and Fair?)

Glo

ba

l In

teg

rity

In

de

xD

e J

ure

an

d D

e F

ac

to

Ind

ica

tors

of

Ele

ctio

ns

In

teg

rity

De Facto

De Jure

Source: Center of Public Integrity, 2006 - http://www.globalintegrity.org/, and Gallup International.

18

Whose Views To Rely On -- Experts or Survey Respondents?

• Expert assessments produced by wide variety of organizations, e.g.

– Multilateral organizations (World Bank CPIA)

– Commercial risk rating agencies (EIU, DRI, PRS, WMO)

– NGOs (Freedom House, Global Integrity)

• Several large cross-country surveys

– of firms (GCS, WEF, BEEPS)

– of households (Afrobarometer, Latinobarometer, Gallup World Poll)

19

Asking the Experts:

• experts natural respondents for certain questions– details of budget or regulatory procedures not

“common knowledge”• are experts “biased”?

ideological biases: do “right-wing experts” like “right wing governments”?: difference between expert assessments and surveys not highly correlated with political orientation of country’s government

– business-elite biases: do businesspeople just want low taxes and no regulation?: business and non-business views on governance often correlate well across countries

• do experts engage in “group-think”? -- in GM V: experts no more correlated among themselves than w/ surveys untainted by group-think, so little evidence of correlated errors

20

Asking Households and Firms• Main advantage is we get information from the ultimate

beneficiaries (victims) of good (bad) governance

– their perceptions matter because they act on them

• Main disadvantages

– costly to do regular cross-national surveys

• concerns about representativeness (but far less than for single-expert assessments)

– badly-designed questions are open to interpretation

• but replace with good questions

• generic problem for all types of indicators

– respondent reticence

• use random response methods to alleviate

-- type of political regime may affect responses (enterprises)

22

-- Thus, triangulation is paramount – looking at various rules and outcome-based indicators, drawn from experts and surveys

-- Further, studying what matters, what is action-worthy is important

23

Anti-Corruption Agencies (GII rating) associated w/ extent of Corruption (as reported by individuals)?

Source: Center of Public Integrity - http://www.globalintegrity.org/, and transparency International.

ARGENTINA

BULGARIA

INDONESIA

INDIA

ISRAEL

KENYA

MEXICO

PAKISTAN

PHILIPPINES

ROMANIA

RUSSIASENEGAL

UNITED STATES

SERBIA

SOUTH AFRICA

0.5

0.6

0.7

0.8

40 50 60 70 80 90 100

Global Integrity Index 2006: "De Jure" & "De Facto" Component of Anticorruption Institutions

Glo

bal

Co

rru

pti

on

Bar

om

eter

200

6:

Fra

ctio

n o

f R

esp

on

den

ts A

nsw

erin

g N

o t

o

"Is

Co

rru

pti

on

Wid

esp

read

?"

r = 0.22

24

Anti-Corruption Agencies (GII) associated with extent of Corruption (as reported by enterprises)?

Source: Center of Public Integrity - http://www.globalintegrity.org/, and World Economic Forum..

ARGENTINA

ARM ENIA

AZERBAIJ AN

BENIN

BULGARIA

BRAZIL

EGY PT

ETHIOPIAGEORGIA

GUATEM ALA

INDONESIA

INDIA

ISRAEL

KENY A

KY RGY ZSTAN

M EXIC O

M ONTENEGRO

M OZAM BIQUE

NIGERIA

NIC ARAGUA

NEPAL

PAKISTAN

PHILIPPINES

ROM ANIARUSSIA

TAJ IKISTAN

TANZANIA

UGANDA

UNITED STATES

VIETNAM

SERBIA

SOUTH AFRIC A

ZIM BABWE

0.2

0.4

0.6

0.8

20 40 60 80 100

Global Integrity Index 2006: "De Facto and De Jure" Components of Anticorruption Institutions

Glo

bal

Co

mp

etit

iven

ess

Su

rvey

200

6:

Co

mp

osi

te C

orr

up

tio

n I

nd

ex

r = 0.02

26

Disaggregate and Aggregate Indicators

• Aggregate indicators have clear drawbacks (but not alone…)• Many individual measures of governance provide imperfect

signals about broader concepts– major advantage of aggregation is that allows reduction

in and transparent quantification of margins of error

• Aggregate indicators are more informative about broad concepts, but at a cost of specificity– average information from distinct sources (WGI, TI)– contrast with “single-source” aggregates averaging

answers from same respondent (e.g. CPIA, GII, DB)• here correlated errors are a major concern

• Not an “either/or” situation: aggregate indicators can be disaggregated

27

Precision and Number of Sources:Control of Corruption, 2006

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0 3 6 9 12 15 18Number of Data Sources

Sta

nd

ard

Err

or

of

Go

vern

ance

Est

imat

e

Source for data: 'Governance Matters VI: Governance Indicators for 1996-2006’, D. Kaufmann, A. Kraay and M. Mastruzzi, (www.govindicators.org);

28

A list of 10 suggestions on ways forward...

1. Many valuable existing ‘baths’, with clean (‘signal’) & some dirty (‘noise’) water. Use & Improve

2. Since we deal w/ proxies: recognize & disclose margins of error in all indicators.

3. ‘Trapping’ Informality (incl. state capture…): let us not think that we solely rely on formal rules

4. Continue to develop “action-worthy” indicators, complementing them with “outcome” indicators

5. Yet, more ‘policy circumspection’: ‘cats & mousetraps’ vs. ‘injections’ -- what Matters is not always ‘WB actionable’

6. Triangulate: do not rely exclusively on any one indicator or source w/out checks/balances w/ othersCont…

29

On Moving forward, cont.7. Match indicator(s)/tool with desired objective: in-

depth country diagnostic, vs. broad country coverage, and exploit complementarities (aggregate & diagnostics; rules & outcomes)

8. Distinguish between 1st vs 2nd order problems and prioritize -- Improving questions, sampling, testing & correcting for biases important (vs. ideology, correlated errors, ‘halo’ effects…?)

9. Perceptions do Matter, & likewise accounting for reports from citizens and entrepreneurs – though margins of error, yet not unique (& judgments galore)

10. Fully disclose data sets (& methods, comparability across countries, periodicity, etc), & subject to rigorous referee review -- & avoid censorship from publication of data

30

Expressions...

Aristotle, Lord Kelvin, Einstein… already covered,

so onto the simple bumper sticker nowadays…

“if you think there are problems with the data,

try without it…”

Lets move forward together…, with some circumspection, cats and mousetraps…