Metrics to Assess the Value of National Market Monitoring, Verification and Enforcement Programmes

Phase I15 September 2017EEDAL Conference, Irvine, California

Nicole KearneyMichael Scholand

Our Mission

CLASP improves the energy and environmental performance of the appliances & equipment we use every day, accelerating our transition to a more sustainable world

3

• Energy & Quality Standards to keep inefficient, cheap products off the market

• Policy Compliance, Testing & Quality Assurance to ensure products perform & markets are fair to all

• Product Labeling & Consumer Education to attract consumers to good products & inspire demand

• Awards & Product Recognition to reward early-movers & accelerate markets

• Procurement, Incentives & Bulk Buys to incentivize innovative manufacturers, reduce risks for all & saturate markets

• Global Collaboration & Knowledge Sharing to leverage cutting edge & collective knowledge and forge productive partnerships

What We Do

Global Standards, Labeling and Compliance Programmes

EFICIENCIA ENERGÉTICARelación de Eficiencia Energética (REE)

determinada como se establece en laNOM-021-ENER/SCFI/ECOL-2000

REE establecida en la norma en ( W/W )

REE de este aparato en ( W/W )

Ahorro de energía de este aparato

IMPORTANTE

Marca: SUPER-IRIS TGV024R200B

860 W

Modelo:

Potencia eléctrica: 17 000 WEfecto neto de enfriamiento:

2,49

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

MenorAhorro

MayorAhorro

El ahorro de energía efectivo dependerá de los hábitosde uso y localización del aparato

Este aparato cumple con los requisitos de seguridad al usuario y no daña la capa de ozono

La etiqueta no debe retirarse del aparatohasta que haya sido adquirido por el consumidor final

REE= Efecto neto de enfriamiento (W)Potencia eléctrica (W)

More than 75 economies have energy efficiency S&L programs in place –Few of these have well-funded compliance programmes

MVE needed to check that products comply to S&L

M - MONITORING - collection and analysis of data, through surveillance of the online and physical market to give a picture of programme progress and compliance

V - VERIFICATION – testing of products using specified test methods to determine whether a product performs according to its claimed energy performance value

E - ENFORCEMENT – action taken in response to non-compliance offences with a suite of timely and appropriate actions; typically based on rigorous testing

S&L Benefits

Safe-Guarding Success…

• Up to 25% of potential S&L programme energy savings are lost through poor compliance and lack of enforcement

• MV&E policies safeguard S&L programme energy savings by ensuring products meet requirements and live up to their energy efficiency claims

“In most markets…

• 20% of the regulated population will comply with any regulation

• 5% will attempt to evade it, and

• the remaining 75% will comply as long as they think that the 5% will be caught and punished.”

- Mark Ellis, Global MV&E Expert

Benefits of MVE

Consumersreceive the expected product at

time of purchase; truth in labels, truth in advertising

Policy MakersAssess S&L programme effectiveness;

achieve key environmental and economic policy objectives

Businessesprovides a level playing field, a fair market

that encourages investment and technological innovation

MV&E Measures Offer Benefits to Key Stakeholders

• Budget must be consistent with programme ambition

• Sufficient staff with the appropriate skills are needed

• Access to expert test laboratories is costly

• Investment in MVE is highly cost-effective

• Costs can be lowered through: o education, clarity of rules, publicity, timely responses to non-

compliance

Provide Adequate Resources for Compliance

Evaluating Impacts

• Many S&L cost benefit and analysis studies and impact assessments conducted to evaluate impacts of appliance efficiency measures around the world and justify further investment.

• Rarely carried out for compliance activities.

Research Objective

• How do country programmes measure the impacts of their compliance policies?

• Are they successfully safeguarding the benefits from S&L programmes?

• What indicators or metrics are used to demonstrate success and justify further compliance funding?

Project Methodology

1. Considered general best practice approach 2. Desk research on programmes around the world 3. Interviews with policy makers from select ‘best practice’ programmes4. Identify best practice evaluation techniques 5. Define and disseminate key metrics, indicators and evaluation

techniques for compliance evaluation to the compliance community

THE CHALLENGEFindings where not what we anticipated, so the project had to shift to a two-stage approach

Monitor and evaluate to continuously improve effectiveness of the programme

• Establish a programme evaluation processo Assess outcomes of compliance activities o Evaluate accountability of programme participantso Develop metrics to assess the performance of the programme (and

justify running costs) Consider lost energy savings for consumers – e.g., 2% more energy in

refrigerators in a market of 500,000 units sold p.a. equates to US$600,000 in lost savings

Look to increase investment in, and develop improvements and effectiveness of the MV&E programme

Monitoring Programme Impact Best Practice

UK Analysis – Quantifying S&L and MVE Benefits

• 2011 UK Study (presented at EEDAL) looked at costs of non-compliance

• Results based on a conservative estimate of 6.2% non-compliance

• Estimated cost of addressing this 6.2% non-compliance: • Total net present value of the cost of non-compliance was £364 million

• Estimated lost benefits from S&L: • CO2 emission savings based on 11 eco-design measures around 7.2 million

tonnes. Applying the estimated 6.2 percent of non-compliance would result in a lost savings of 0.4 million tonnes annually in the UK by 2020.

Ghana

Country Perspectives on Measuring MVE Impacts

Interviews in 4 “best practice” countries,

with robust and different compliance

programmes

Country Interviews

• Aim to identify:

• Key success factors or attributes of the compliance programme

• How they secure compliance funding

• Whether or how they track the effectiveness of this work

• Do they apply quality assurance metrics to track performance and demonstrate value of compliance programme to the national economy

• Ultimately, how they justify investment in their compliance programme

Case Study: Australia

• Key attributes • High investment in the programme, which

involves regional coordination with New Zealand • Pre-market registration requirements• Combination of state and national level compliance support • Inspectors with inspection, monitoring and investigation powers • Since moving to a national level programme, more investment and activities • Strong market surveillance to gather data and identify non-compliance

• Funding received from:

• Product registrations and government

• NO evaluation or performance metrics in use - but…• Product registration database allows policymakers to track compliance and

efficiency levels • Market surveillance activities to gather data

Case Study: Ghana

• Key attributes• Import market • Third party verification required • Most compliance checks carried out at customs • Shared responsibility between government agencies• Strong communications efforts to support compliance

• Funding received from: • UNDP Project • Cost of the enforcement action (including testing and staff time) is

charged to the supplier

• NO evaluation or performance metrics in use - but…• Monitoring surveys to engage with consumers and retailers on

energy efficiency and the importance of compliance

Case Study: Sweden

• Key attributes• Collaboration with Nordic market surveillance authorities • Risk based and targeted approach to compliance checking • Strong focus on communications to support industry • Government testing facilities in use

• Funding for compliance from: • Government budget • Financial penalties associated with non-compliance

• NO evaluation or performance metrics in use - but…• Participants in an EU Commission Task Force looking into evaluating MV&E• Conducting their own assessment based on follow-up with market

surveillance plans. For example, which activities in the plan were carried out and which were not and why.

Case Study: United Kingdom

• Key attributes• Shift from local to national market surveillance based on

impact assessments and research • Collaboration across government on market surveillance

activities and approaches to non-compliance • Risk based and targeted approach to non-compliance • Combination of verification testing and screen-testing of products • Strong dialogue and collaboration with industry

• Funding for compliance from: • Government budget

• NO evaluation or performance metrics in use - but…• Informal mechanisms to calculate effectiveness of their work • Annual reporting • New approach being explored

One common theme…

NO evaluation or measurement for national compliance programmes

Indicators to measure impacts of MVE

Specific, measurable, reliable and simple

• Assess progress toward a specific outcome or goal

• Inform whether the market is transitioning or not

• Track indicators from: • Outset of programme - input indicators to inform baseline

• During the programme – output indicators to evaluate impacts

Indicators to measure impacts of MVE

Figure 1. Example of some key indicators that are contained in the stages of a MVE scheme

Project Conclusions

• Very little evidence of compliance evaluation

• Evaluation needed to justify compliance spend around the world

• No clear metrics or indicators in use for measuring compliance

• Statistics for non-compliance are very complex

• Need to conduct a second phase of the project to define indicators, metrics, and evaluation processes for compliance

Thank you! Nicole Kearney – nkearney@clasp.ngo

Mike Scholand – mscholand@clasp.ngo