Key issues of measuring the effectiveness of compliance

assurance

Eugene MazurOECD Environment Directorate

Why measure outcomes?

To know that policies and laws are actually implemented!

• Meeting legal and policy obligations• Internal management needs• Budget justification• Demand for enhanced external

accountability

The traditional focus on input and output measures is not enough to assess effectiveness!

2

What to measure?

Intermediate outcomes – changes in compliance knowledge & behaviour of the regulated community

• Compliance/noncompliance rates• Measures of recidivism/duration of

noncompliance• Pollution release indicators• Indicators of improved environmental

management practices and reduced risk• Measures of effectiveness of compliance

assistanceFinal outcomes – improvements in

environmental conditions as a result of compliance assurance

• Environmental quality indicators

3

How good are the indicators?

• Measurability – available at a reasonable cost, regularly updated via a reliable procedure

• Analytical soundness – well founded in technical and scientific terms– Scope definition, statistical validity, observation

periods, link between activities and outcomes• Policy relevance – user-friendly, easy to

interpret, provides a basis for international comparisons

• Phase 1 OECD project (2009-2010): analysis of practices in 10 countries

4

Assessment matrix

5

Measurability Analytical soundness

Policy relevance

Compliance rates M L M

Recidivism rates M L M

Pollution releases H M H

Improved management L L M

Effectiveness of assistance L L M

Environmental quality H M M

A toolbox – impossible to identify a “best practice” approach or a set of “flawless” indicators.

Key implementation principles• Targeting outcome indicators on concrete

regulatory priorities (pollutants, sectors) improves analytical soundness but reduces comparability– Develop compliance rates for specific sectors and

regulations– Focus pollution release indicators on priority

pollutants, use number of pollution incidents– Use only environmental quality indicators that

correspond to targeted, problem-oriented efforts• Link outcome indicators with time-specific

targets• Trends analysis in conjunction with input,

output indicators greatly increases policy relevance

6

Main issues going forward

• How to make compliance rates more meaningful?

• Using composite indices to characterise compliance outcomes?

• How many compliance indicators to measure (costs and benefits)?

• Is it feasible to develop a core set of indicators comparable nationally (decentralised systems) and internationally?

Phase 2 OECD project (2013-2014)

Compliance rates

Two main problems with compliance rates:

• Statistical bias as a result of ever-increasing emphasis on risk-based targeting

• Binary nature (in or out of compliance)– Differentiate non-compliance based on the

potential seriousness of environmental impact

8

Composite indicators• Despite the cost and misleading nature of

composite indicators (indices), they continue to be developed:– Perceived usefulness as a communication tool– Measure effectiveness on a scale

• Essentially, three categories of compliance-related indices:– Pollution release indices, for priority pollutants,

weighted for their relative harm (NSW, Australia)– Weighted compliance rates to distinguish

between procedural and substantive violations (Ontario, Canada)

– Risk scores: about more than compliance, but allows a comparable indicator of number of high-risk facilities

9

Number of indicators: costs and benefits

• Key challenge: how to measure time/cost requirements for data collection of certain types of outcome indicators

• Prefer intermediate outcome measures with the highest “measurability” (e.g. pollution releases, incidents)

• Pick those most closely linked to input and output indicators

• Promising “value for money” projects in the UK and Ireland

10

Comparability

• Designing comparable indicators not analytically impossible but politically difficult

• Primary need – in decentralised systems of environmental governance (level playing field across sub-national jurisdictions)

• Some “candidates”:– Rate of serious non-compliance for 3-4

“marker” requirements (e.g. linked to EU Directives)

– Number of high-risk installations (top score band)

– Number of serious pollution incidents per number of high-risk installations

– Monetary value of complying actions

11