Participación del Public Utility Research Center (PURC), en la

V Reunión Anual de ADERASA

Quality Indicators of the Regulatory Process

Maria Luisa CortonGramado,

Rio Grande do SulBrazil

3 y 4 de Octubre de 2005.

Topics:1) Regulatory Process Quality Indicators

2) How to Recognize Good and Bad Quality Regulation

3) Regulatory Leadership: Resolving Conflict and Making Choices

4) Dimensions of Performance:

5) How to establish empirically the role regulatory decisions haveplayed in sector performance outcomes

6) Recent work on Regulatory Quality Performance

7) What are we doing at PURC

This presentation draws from

“Crisis, Concern, and Complacency: Managing Conflict in Water Supply,”by Sanford V. Berg (PURC)

A Manual for Evaluating the Effectiveness of Infrastructure ReguA Manual for Evaluating the Effectiveness of Infrastructure Regulatory latory Systems, Systems, forthcoming World Bank book by Ashley Brown, Jon Stern and Bernard Tenenbaum

“Regulatory Effectiveness: the Impact of Regulation and Regulatory Governance Arrangements on Electricity Industry Outcomes,” Stern and Cubbin, World Bank Policy Research Working Paper 3536 (March 2005)

“Leadership and the Independent Regulator,” by Mark Jamison, World Bank Policy Research Working Paper 3202 (June 2005)

The performance of the sector is the ultimate indicatorultimate indicator of Regulatory Performance

1. Regulatory Process Quality Indicators1. Regulatory Process Quality Indicators

Productivity advance: reflecting cost containment and adoption of new technologies

New service introductions (example, wastewater treatment)

Returns to investors commensurate with the risks they bear (where national development banks and Ministries of Finance are investing on behalf of citizens)

Prices that reflect minimum optimum costs

Expansion of basic services to particular customer groups

2.How to Recognize 2.How to Recognize Good and Bad Quality Regulation?Good and Bad Quality Regulation?

Good Regulation Good sector outcome

Bad Regulation Bad sector outcomeAlways

?

This is not about assigning blame for mistakes

1. Inability to enforce regulatory decisions (due to lack of legal authority)2. Inappropriate process for reviewing regulatory decisions (weak legal

system)3. Government (political) interference and/or usurpation of authority4. Inadequate power or discretion granted to regulator and other

agencies or institutions5. Constitutional or other legal constraints 6. Adverse macro-economic circumstances, currency problems, fiscal

or monetary policies7. Lack of adequate professional, technological or financial resources8. Perception of overall country risk by private investors9. Natural or other disasters or shortages of essential resources

2.1 Some Influential factors 2.1 Some Influential factors not controllablenot controllable by the by the RegulatorsRegulators

2.2 External & Internal Factors Affecting Outcomes2.2 External & Internal Factors Affecting Outcomes

Macro economic conditions

Sector OutcomesSector Outcomes

Currency fluctuationsInterest rates Management skills

Political and social conditionsRegulatory Processes and governance

Regulatory Decisions

Market structure

2.3 Bad Outcomes Arising from Regulatory Inaction: 2.3 Bad Outcomes Arising from Regulatory Inaction: Sins of OmissionSins of Omission

• Failure of systematically collect and effectively analyze all information required for regulatory decisions

• Inadequate or absence of auditing regulated enterprises

• Insufficient or inadequate definition of regulatory methodologies in place

• Lack of setting adequate standards where needed• Failure to consider public feedback and critiques of

regulatory decisions

2.4 Bad Decisions Taken by Regulators: 2.4 Bad Decisions Taken by Regulators: Sins of CommissionSins of Commission

• Light penalties assessed for serious deficiencies (or alternatively)

• Using inaccurate data or misapplying accurate data• Providing perverse incentives (encouraging over

investment or discouraging service quality)• Using inadequate methodology (price, cost of

capital, financial sustainability)• Inappropriate differentiation of different classes and

type of customers

2.5 Why Is It So Difficult to Evaluate the Quality of 2.5 Why Is It So Difficult to Evaluate the Quality of Regulatory Decisions? Regulatory Decisions?

Regulatory Decisions Generally have Winners and Losers

Conflicts

3. Regulatory Leadership: Resolving Conflict & Making 3. Regulatory Leadership: Resolving Conflict & Making ChoicesChoices

Utility will spend $700,000 in OPEX.

With $300,000 in CAPEX, the utility claims it can provide

3000 new connections OR

Produce an increase of “six points”in the water quality index, OR

Be at another point on the Frontier (A, B, or C).

Change in Index of Water Quality

NumberOf NewConnections

5000

4000

3000

2000

1000

1 2 3 4 5 6 7 8 9

A

B

C

Regulator: Source of Conflict?Regulator: Source of Conflict?

1.Authority Conflicts: lack of clarity of roles

2.Cognitive (Factual) Conflicts: disagreements regarding current or historical facts and causal linkages

3.Value Conflicts: conflicting priorities and weights on outcomes

4.Interest Conflicts: stakeholders benefit differentially from decisions

3.1 Authority Conflict3.1 Authority Conflict

“Authority” conflicts” reflect different views regarding wheredecisions will or ought to be made.

Jurisdiction may not yet be assigned or the issue might be addressed by multiple agencies. Stakeholders will go jurisdiction-shopping—selecting the agency or the level of government most likely to support its interests in policy design and implementation. Appeals procedures within the judicial system can delay implementation. In such situations, benefits delayed are (effectively) benefits denied.Issues include: social justice (or fairness) and environmental impacts

Authority Issues Facing RegulatorsAuthority Issues Facing Regulators

Who decides where to expand networks?Who funds network expansion?Who determines when prices are financially sustainable?

Who monitors water quality?Who sets water quality standards?Who decides environmental requirements?Who makes water resource allocation decisions?

Change in Index of Water Quality

NumberOf NewConnections

5000

4000

3000

2000

1000

1 2 3 4 5 6 7 8 9

Regulator: Conflict ResolutionRegulator: Conflict Resolution----AuthorityAuthority

Seek Changes in the Law—legal clarity

Cooperate with Sister Agencies (avoid turf wars)

Establish Task Forces to Address Issues

Educate the Courts

Improve Appeals Procedures

3.2 Cognitive Conflicts3.2 Cognitive Conflicts“Cognitive” conflicts are disputes over factual matters: “What is?”

For example, How many new connections can be made with $300,000?

Technical disagreements reflect cognitive conflicts. Such conflicts can be reduced through comprehensive data collection and analysis.

Factual Issues Facing RegulatorsFactual Issues Facing Regulators

With $300,000 in CAPEX, Citizens claim the utility can provide

5000 new connectionsOR

Produce an increase of “10 points”in the water quality index,

ORBe at another point on the Frontier (D, E or F).

Change in Index of Water Quality

NumberOf NewConnections

5000

4000

3000

2000

1000

1 2 3 4 5 6 7 8 9

A

B

C

D

E

F

Regulator: Conflict ResolutionRegulator: Conflict Resolution----FactsFactsBenchmarking Studies: – Input Data (physical and monetary)– Output Data (connections, water delivered, continuity,

quality)

Financial Sustainability Studies – Income Statements– Balance Sheets– Cash Flow Statements

Examine Incentives and Estimate time to reach “frontier”

3.3 Values Conflicts3.3 Values Conflicts

“Values” conflicts are more ideological in nature, reflecting the different preferences or values of groups.

Is there a political consensus over the weight assigned to particular outcomes, especially outcomes involving non-monetary impacts?

Targets: Preferred outcomes depend on citizen attitudes.

Values Issues Facing RegulatorsValues Issues Facing Regulators

Once on the FrontierShould more than $300,000 be invested?

Which Target: D? E? F?Who sets the Target?

Change in Index of Water Quality

NumberOf NewConnections

5000

4000

3000

2000

1000

1 2 3 4 5 6 7 8 9

D

E

F

Regulator: Conflict ResolutionRegulator: Conflict Resolution----ValuesValuesPublic Education– Publish Performance Comparisons– Identify Trade-offs– Reports to the Legislature

Promote Citizen Participation – Talk Radio– News Conferences– Citizen Advisory Boards

Limit the Rhetoric, but Articulate a Vision

3.4 Interest Conflicts3.4 Interest Conflicts“Interest” conflicts reflect the differential impacts of policies on various stakeholder groups: “For whom is the policy?”If the situation is actually a zero-sum game, one group benefits at another’s expense (unless there is compensation). Special Interests: The political economy of regulation suggests that when the beneficiaries of a particular policy are concentrated (and per capita benefits are high) the beneficiaries will lobby.If losers are diffuse (and the per capita damages are low), the result is a policy that benefits well-organized stakeholders—even when the costs to the losers outweigh the benefits to the winners.

Special Interest Issues & RegulatorsSpecial Interest Issues & Regulators

Should more than $300,000 be invested?What should prices be to different groups?

Unserved citizens want “D”

Current Customers want “F”

Which Group has Political Power?

Change in Index of Water Quality

NumberOf NewConnections

5000

4000

3000

2000

1000

1 2 3 4 5 6 7 8 9

D

E

F

Regulator: Conflict ResolutionRegulator: Conflict Resolution----InterestsInterestsDo not pretend there are no conflicts– View from the Balcony—step back from stakeholders

Take a Leadership Role in Identifying Benefits and Costs– OFWAT and EU Environmental Standards– Collected compliance cost information from utilities– Presented costs of meeting targets: current deadline

vs. delay

Politicians make Final Decision

Informal Survey: Consider how you would answer the Informal Survey: Consider how you would answer the following:following:

1. Authority: Is the regulatory agency used to help resolve key issues or is it by-passed? Are you in the middle of turf wars?

2. Facts: Has benchmarking been used to improve sector performance through? Are contracts and targets based on reality?

3. Values: Does the regulatory agency help clarify the goals represented by targets or stated political objectives?

4. Special Interests: Have regulatory decisions been inconsistent due to the influence of special interests?

4 Dimensions of Regulation: Formal & Informal4 Dimensions of Regulation: Formal & Informal

Formal elements1.Clarity of roles and objectives2.Autonomy3.Accountability

Informal elements4.Participation5.Transparency6.Predictability

Stern & Holder (1999) framework for appraising Regulatory Systems

4.1 Dimensions of Regulation: How & What4.1 Dimensions of Regulation: How & WhatHow Governance: Institutional and legal design

•Independence and accountability•Relationship between the regulator and policy maker•Processes by which decisions are made•Organizational structure and resources available to the regulator

What Substance: content•Tariff levels and structure•Quality of service standards•Classification of customers•Handling consumer complaints

Regulatory System = Institutions + Laws + Processes

4.2 Evaluating Performance4.2 Evaluating Performance

1. Criteria for evaluating Regulatory Process and Governance

2. Criteria for judging Regulatory Decisions

Quality of decisions themselves

Factors that cause them

4.3 Processes and Outcomes4.3 Processes and Outcomes

Formal legal power and attributes

Regulatory Processesand Governance

Regulatory Decisions

Means by which regulatory agencies Interpret their power and duties

Output of the SystemOutput of the System

+ Regulatory Substance

+

Any evaluation of the regulatory System will be incomplete if it is Limited only to regulatory process and governance

5. How to establish empirically the role regulatory 5. How to establish empirically the role regulatory decisions have played in the sector performance?decisions have played in the sector performance?

Possible weaknesses when performing evaluation Focus on the institutional and legal characteristics

Lack of follow up on legal requirements implementation

Absence of considering Regulatory decisions effect

Attempt to quantify the impact of regulation on sector performance requires substantial resources and robust methodologies

5.1 From Managing Conflicts to Promoting Best 5.1 From Managing Conflicts to Promoting Best PracticePractice

Regulatory Principles Regulatory Principles

Institutional and Institutional and Legal characteristicsLegal characteristics

Independence

Management

FinancialDecision Making

The point of any evaluation should be to provide the basis for learning

Any evaluation requires to consult the widest possible range of affected parties

First task for evaluation is to define current state of the sector

Data from several years is needed

5.2 Baselines, Benchmarks, and Progress5.2 Baselines, Benchmarks, and Progress

EVALUATI

ON

BENCHMARKI

NG

Self Examination: risk of denying symptoms That are obvious to everyone but the regulator

5.3 Evaluating Decisions5.3 Evaluating Decisions

Evaluations should be performed by those whose judgment and recommendations are Perceived as fair and objective

Inside and Outside the country

Account for different types of regulatory entities

5.4 Relating Regulatory Outputs to Sector Performance5.4 Relating Regulatory Outputs to Sector Performance

Collecting data on the following indicators allow a judgment on the role of the regulatory framework in achieving improved sector outcomes

1. Output and Consumption2. Efficiency3. Quality of Supply4. Financial Performance5. Capacity, Investment and Maintenance6. Prices7. Competition

5.5 Recent work: Returns and Risk5.5 Recent work: Returns and RiskThe paper examines investors returns on infrastructure and its adequacy to therisk taken.

It analyses the impact of “quality of regulationquality of regulation” on this adequacy.

An index on quality regulation composed of– Legal solidity, – Financial strength and – Decision making autonomy.

by Sirtaine, Pinglo, Guash and Foster “ How profitable are private infrastructure concessionsin Latin America? Empirical Evidence and Regulatory implications–WB andQuarterly Review of Economics and Finance, 2005.

5.6 Measures of Regulatory Quality used in the study 5.6 Measures of Regulatory Quality used in the study Legal solidity: predictability of the regulatory regime– Regulatory framework established by law

Financial strength: resources to undertake its functions– Source of financing– Regulatory budget as a percentage sectoral GDP

Decision making autonomy: – Independency of appointment– Duration of appointment– Individual Regulator or Commission (collegiality of

decisions)

5.7 Study of Regulatory Quality Performance5.7 Study of Regulatory Quality Performance

34 concessions over 7 years of data for Argentina, Brazil, Bolivia, Chile, Colombia, El Salvador, Mexico,Peru, Venezuela

Estimate historical and projected returns of investors and concessions

Shareholder IRRReturns on Equity Cost of Equity

Project IRRReturn to Capital

WACC

5.8 Possible Results 5.8 Possible Results

Shareholder IRR > CE : Shareholders have earned excess returnscompared to the risk taken

Return on Equity > CE : Concession has returned post tax profitability superior to alternative investments of similar risk

Project IRR > WACC : Concession has generated positive net financial flows

Return on Capital > WACC : Concession net operating profitability exceeds the level necessary to service its debt and equity

5.9 Key Findings of the study5.9 Key Findings of the study

Financial returns of private infrastructure concessions have been modest and some even below the cost of capital

On average telecom and energy perform better than transport and water

There is a considerable variance across concessions explained mainly by the quality of regulation

7. What are we doing at PURC7. What are we doing at PURC

Excellent Source of Information: Regulatory Body of Knowledge

Literature Survey

Links to Glossary

Links to Papers

Links to self-paced Testing

7.1 Water Benchmarking Support System7.1 Water Benchmarking Support SystemThe Public Utility Research Center is funded by the World Bank tThe Public Utility Research Center is funded by the World Bank to o prepare a Water Benchmarking Support System that will include a prepare a Water Benchmarking Support System that will include a Survey of Benchmarking Methodologies and a Benchmarking Check Survey of Benchmarking Methodologies and a Benchmarking Check List. List.

These will become part of a new Benchmarking Toolkit for regulators around the world, and included as an IBNET resource. PURC will work closely with ADERASA Benchmarking Task Force on the technical portion of the report addressing the strengths and limitations of different methodologies for comparing performance trends over time and forranking utilities.

We need to describe the extent to which regulatory commissions are currently collecting and analyzing data.

Please, take some time to answer the survey handed in to you

4) Contactos:

Director of Water Studies:sanford.berg@cba.ufl.edu

Research Associates:chen.lin@cba.ufl.eduguillermo.sabbioni@cba.ufl.edumcorton@ufl.edu

Página Web:

http://www.purc.org

Trabajos en el sector agua:http://bear.cba.ufl.edu/centers/purc/conferences/water.htm

Trabajos en español:http://bear.cba.ufl.edu/centers/purc/publications/04.htm

Contactos:

Los esperamos en el próximo Programa de Entrenamiento Internacional en Regulación de Servicios Públicos, PURC - Banco Mundial.

Del 9 al 20 de Enero de 2006

Gainesville, Florida

Participación del Public Utility Research Center (PURC), en la

V ReuniV Reunióón de ADERASAn de ADERASA

Maria Luisa CortonGramado, Rio Grande do Sul

Brazil3 y 4 de Octubre de 2005.

Muito Obrigado!Muchas Gracias!