LESSONS LEARNED - Stanford University...¥Coordination between energy efficiency advocates,...

1

LESSONS LEARNEDLESSONS LEARNED AFTER 30 YEARS AFTER 30 YEARS

OF PROCESS EVALUATIONOF PROCESS EVALUATION

Funded By:

Presented By:

Jane S. Peters, Ph.D.

11/09/07 LESSONS LEARNED AFTER 30 YEARS OF PROCESS EVALUATION 2

2004-2005 Statewide Building Operator Certification and Training Program: MPER #3

BackgroundBackground

!Energy efficiencyprograms beganin the mid-1970s

!Evaluation ofthese programsbegan soonthereafter

2




!Evaluation, especially processevaluation, has been an importanttool in improving program designand implementation




Today,30 years later,

what do weknow??

3



How Do We Know What We KnowHow Do We Know What We Know

!Process evaluation results are rarelyreported in conferences

!Process evaluations are rarely published

!Process evaluators and their clientstend to retain the knowledge



How Do We Know What We KnowHow Do We Know What We Know

!Therefore:• Interviews with

processevaluators are alikely source oflessons learned

4



Interviews with 18 Respected EvaluatorsInterviews with 18 Respected Evaluators

2

11

5

0

2

4

6

8

10

12

Internal External Both

Type

8

3

5

0

20

1

2

3

4

5

6

7

8

> 25 yrs 20-24 yrs 15-19 yrs 10-14 yrs < 10 yrs

Experience



Topic Areas in White PaperTopic Areas in White Paper

!Program design and implementation• Technologies, rewards and incentives, and

implementation

!Program administration• Databases, program processes

!Reaching market actors• Trade allies, professional services

providers, manufacturers and vendors

5



Topic Areas in White PaperTopic Areas in White Paper

!Reaching the customer• Communication, residential customers,

businesses and industry

!Leaning from evaluation• Going beyond satisfaction, benefits and

timing



Program Design and ImplementationProgram Design and Implementation

Technologies

6



TechnologiesTechnologies

!What do these have in common?• High efficiency motors

• T-8 lamps and electronic ballasts

• Compact florescent lamps

• Passive solar homes



TechnologiesTechnologies

!All had many reliability problems atintroduction

!Many early adopters resisted lateradoption due to bad experiences

!Resistance continues for some

7



Types of Technology ProblemsTypes of Technology Problems

!Transferring passive solar designs fromone region to other regions• Storage requirements vary

• Room configurations vary

!Efficient motors and T-8s• Poor performance

• Poor reliability

• Not cost justifiable



Types of Technology ProblemsTypes of Technology Problems

!CFL problems are legendary• Light quality• Start time• Price• Fit with existing fixtures• Variability• Plethora of pin types• Manufacturing irregularities

8



Types of Technology SolutionsTypes of Technology Solutions

!What is different about high efficiencyclothes washers?• The High Efficiency Laundry Metering and

Market Analysis project (THELMA)– Market and process evaluation research

– Impact research

• Coordination between energy efficiencyadvocates, utilities, state agencies, andmanufacturers



Types of Technology SolutionsTypes of Technology Solutions

!The result is a product that has highmarket acceptance and manufacturersupport

9



Program AdministrationProgram Administration

ProgramProcesses



Types of Program Process ProblemsTypes of Program Process Problems

!Doing energy efficiency is actuallydifficult

!Decisions about what is efficient• Where to get the information

• Sorting through different claims

!Convincing trade allies you really dowant the efficient solution

!Completing forms about stuff you knownothing about – over and over again

10



Types of Program Process SolutionsTypes of Program Process Solutions

!Simpler processes• On-line forms• Check boxes

!Use trade allies to complete forms

!Payment in a timely fashion (days notweeks)

!Reasonable inspection practices

!Coordination with others offeringincentives• Tax credits



Reaching Market ActorsReaching Market Actors

TradeAllies

11



Trade AlliesTrade Allies

!Trade allies are the plumbers,contractors, electricians and buildingtrades people who install energy-usingequipment

!They can make a program work well ornot work very well



Working with Trade AlliesWorking with Trade Allies

!Changes to programs affect trade allies• Too frequent hurts their ability to make

commitments to customers

• Without consulting them, they are notprepared and can’t plan and orderequipment

!Trade allies who receive the incentive• Are more willing to complete paperwork

• Are more willing to complete it correctly

12



Working with Trade AlliesWorking with Trade Allies

!Inspections and quality assurance areneeded• Technical requirements are often new

• Trade allies will try to leverage incentives



Reaching the CustomerReaching the Customer

ResidentialCustomers

13



Reaching Residential CustomersReaching Residential Customers

!This area was the focus of manyprocess evaluations in the 1970s - 1980s

!Remaining savings are smaller thanfrom business and industry andtransaction costs are high



Reaching Residential CustomersReaching Residential Customers

!Barriers are high• First cost

• Awareness, knowledge, access

!Interest is also high• Energy efficiency important for more than

80% of the population since the 1980s

14



What Works with Residential CustomersWhat Works with Residential Customers

!Decision events are where opportunitieslie• Remodeling a home

• Buying a new or existing home

• Buying a replacement for a broken waterheater or dishwasher, etc.



What Works with Residential CustomersWhat Works with Residential Customers

!Help consumers know what is energyefficient• Consumers think that everything already is

energy efficient

• Consumers can’t tell how much benefit is tobe gained from choosing a SEER 14 over aSEER 12

• ENERGY STAR® labeling

15



Learning from EvaluationLearning from Evaluation

GoingBeyond

Satisfaction



Going Beyond SatisfactionGoing Beyond Satisfaction

!Process evaluations are not just aboutcustomer or market actor satisfaction

!The purpose should be to understandwhy there is satisfaction ordissatisfaction and what can be done tochange the latter• High satisfaction might mean the program is

not good for the ratepayers

• Dissatisfaction might mean the program justneeds a little tweak

16



Lessons LearnedLessons Learned

FiveLessons



Five LessonsFive Lessons

!Conduct research on new technologiesprior to mass introduction

!Simplify! Simplify! complexity reducesparticipation

!Trade allies are key to program success

!Targeting decision-points reducestransaction costs to reach consumers

!Going beyond satisfaction is whereopportunities for improvement are found

17



WhatWhat’’s Next?s Next?

This Is JustThe

Beginning



This Is Just the BeginningThis Is Just the Beginning

!These are just five of the 13 sections ofthe White Paper

!There are more lessons• Interviews with additional experienced

evaluators

• Review of the literature identified by thoseinterviewed

• Opportunities with future BECC to expandon lessons learned

18



Contact InformationContact Information

Jane S. Peters

[email protected]

Paper and presentation available atwww.researchintoaction.com

LESSONS LEARNED - Stanford University...¥Coordination between energy efficiency advocates,...

Documents

Transcript of LESSONS LEARNED - Stanford University...¥Coordination between energy efficiency advocates,...