Evaluating Renewable Energy Programs

A Program Manager’s Guide July 12, 2011

Warren LeonClean Energy States Alliance

1

Housekeeping

2

You will be connected to audio using your computer’s microphone and speakers (VoIP) or through your headset.

If you are calling into the webinar, please select “Use Telephone” on the webinar’s control panel after joining the Webinar. Make sure to enter your Audio PIN, shown in the control panel where you choose the option to join by telephone.Dial: 1 (805) 309-0022Access code: 381-923-136Webinar ID: 176-245-041Audio PIN: Shown after joining the meeting

You will automatically be muted when you join the webinar. We’ll take questions after all the presentations have been made; you can queue up a question anytime for the organizer to ask at the end of the session by typing it in the Question box of the control panel. We can also unmute you to ask your question if you raise your “hand.”

This webinar is being recorded and will be made available after the call on the CESA website at www.cleanenergystates.org in the Members section of the website.

No monthly call in August MOUs for 2011-2012 have been mailed.

Please contact Maria or Anne with any questions at 802-223-2554.

We will be sending out a preliminary agenda and pre-registration for the October 24-26 members meeting in Washington, DC, shortly. Please pre-register ASAP!

Reminders

3

Explain what is in the report

Make it easier for you to use the report

Cover some of the recommendations in the report

Answer your questions and/or discuss whatever you are interested in

Purpose of the Webinar

4

Considers evaluation from the perspective of the program manager

Seeks to help you identify evaluation activities that will be useful, cost-effective, and well-received by program staff, policymakers, and stakeholders

Considers how to choose among and approach different types of evaluations

Serve as a reference guide: designed to make it easy to find the sections relevant to you

Purpose of the Report

5

How to prepare for effective evaluation◦ Setting program goals

◦ Producing a program theory

◦ Setting evaluation goals

◦ Choosing an evaluator

Discussion of five types of evaluation, with recommendations for how to approach each

Framing and presenting evaluations for external audiences

The Report Contents

6

Developing a program theory◦ From California Evaluation Framework

Sample program logic model◦ From NYSERDA

Possible evaluation questions◦ From EERE Guide to Managing General Evaluation

Studies Descriptions of models used in cost-benefit

evaluations: IMPLAN, JEDI, REMI Reference works Representative evaluation reports

Appendixes

7

My Starting Point

Evaluation: An Essential Part of Good Program Management1. Improve the quality and efficiency

of programs

2. Put a program into context by helping managers & stake-holders understand what it is accomplish-ing and how it compares to other programs

3. Demonstrate that the agency takes its responsibilities seriously and seeks to maximize the public benefits of public spending

8

Evaluation doesn’t receive enough attention Few widely accepted protocols for

evaluating renewable energy programs Difficult to evaluate the results of some

programs Program goals may not be explicit or fully

thought out The decision about what to evaluate is not

made by program managers

Why Are Some Evaluations Unsuccessful?

9

“Not everything that counts can be counted, and not everything that can be counted

counts.” Albert Einstein

Evaluation Can Be Difficult

10

1. Needs and Market Assessments

2. Process Evaluations 

3. Outcome Evaluations ◦ Some evaluators use “gross outcomes”

4. Impact Evaluations ◦ Some evaluators use “net outcomes”

5. Cost-Benefit Evaluations

Types of Evaluations

11

What It Does◦ Identifies target markets ◦ Identifies barriers to the adoption of renewable energy ◦ Understand a market or audience

Why It Is Used◦ Help program managers design programs ◦ Establish baselines for measuring future progress

Key Comments◦ They can lead to better programs◦ They can create baselines for future evaluations of

program outcomes and impacts◦ Clean energy agencies should do them more

frequently

Needs and Market Assessments

12

What It Does◦ Examines program implementation

processes and operations

◦ Determines whether the program is well-designed, efficiently managed, and effectively marketed

◦ Assess customer satisfaction

Why It Is Used◦ Identify ways to improve the program

◦ Understand the views of customers and other stakeholders

Process Evaluations

13

They come in many shapes and sizes

Renewable energy agencies should do them more frequently—especially small, focused evaluation reports

Customer satisfaction surveys are especially valuable

Comments on Process Evaluations

14

What It Does◦ Determines whether the program

is achieving its intended outcomes and objectives

Why It Is Used◦ Keep program managers and

others focused on the program’s goals

◦ Know whether a program is achieving its objectives

◦ Determine whether the program should be modified so that it is better achieving its objectives

Outcome Evaluations

15

It is important to know if a program is achieving its goals

Depending upon the program and its intended outcomes, an outcome evaluation can be relatively straight-forward or very complicated

Consider up front whether the dissemination of research findings can help key stake-holders and will move a program towards its goals

Comments on Outcome Evaluations

16

1. Energy outcomes

2. Environmental outcomes

3. Electricity system outcomes

4. Economic outcomes

EPA report: Assessing the Multiple Benefits of Clean Energy: A Resource for States

For renewable energy installation programs: easier to quantify their energy & environmental outcomes than their electric system & economic outcomes

Installation Programs: Four Types of Outcomes

17

Market transformation programs Business development programs Research, R&D, and demonstration

programs Education and information programs

Outcomes of Other Types of Programs Need to Be Evaluated in Different Ways

18

What It Does◦ Determines the share of the outcomes caused by

the program rather than other factors

◦ Identifies unintended but valuable benefits of the program

Why It Is Used◦ Understand what the program is actually causing

to happen

◦ Determine whether the program is unnecessarily providing funding to free riders who do not need the program to act

Impact Evaluations

19

The more complicated the program theory and the more multi-faceted the program’s route to achieving outcomes, the harder it is to determine which outcomes would have occurred without the program◦ Consider how difficult it will be to determine your

program’s impacts and how precise an answer you need

It can be helpful to gather the results from evaluations of similar programs in other states◦ If the evaluation yields results significantly different than

other states, have the evaluator provide an explanation

Choose experienced evaluators Piggy-back other questions onto a survey

Comments on Impact Evaluations

20

What It Does◦ Compares the economic and/or other benefits of a

program’s impacts to the cost of achieving those impacts.

Why It Is Used◦ Determine the extent to which the program’s

benefits outweigh its costs

◦ Understand whether the program is cost-effective

◦ Decide whether the program should be continued as is, modified, or ended

Cost-Benefit Evaluations

21

Proceed cautiously These evaluations usually

involve predictions about the future, introducing considerable uncertainty

Consider indirect costs and benefits

Choose experienced evaluators & understand their methodology

Make sure the presentation of results reveals rather than obscures the assumptions and uncertainties

Comments on Cost-Benefit Evaluations

Consider low-cost alternatives

• Commentary by an economist• Comparisons between states

22

1.Think carefully about why you want to undertake the evaluation

2.Choose an experienced evaluator (including ones with pre-existing data & models)

3.Make sure you understand the research methods

4.Have the evaluator justify key assumptions

5.Assess more than one scenario

6.Understand the discount rate

7.Ask the evaluator to examine comparable studies & justify differences

8.Understand exactly what the results show

9.Make them comparative10.Don’t imply the results

are more accurate than they are

A Ten-Step Approach for Cost-Benefit Evaluations

23

1. Identify all the audiences for the study

2. Make sure the evaluators understand the audiences

3. Develop an outreach plan

4. Decide whether the evaluators should produce collateral material

5. Don’t present the results with inappropriate precision

6. Decide on responses to the report and what to share

7. Disseminate relevant information to stakeholders

A Strategy for Communicating with External Audiences

24

Warren LeonSenior Advisor

Clean Energy States Alliancewleon@cleanegroup.org

www.cleanenergystates.org

Questions and comments?

25