Comprehensive Usage Reduction Program Evaluation

Comprehensive Usage Reduction Program Evaluation

Affordable Comfort 2007

Jacqueline Berger

Evaluation Objectives

1. Determine the cost-effectiveness of WRAP

2. Develop standard questions so PPL can measure the same criteria in future evaluations

3. Comply with the PUC Order

2

Evaluation Questions

1. What are the goals and how are they met?

2. What are the admin costs? How can they be lowered?

3. How effective is program recruitment?

4. Is the audit mechanism effective?

5. Is the list of program measures comprehensive?

3


6. Is the education process cost-efficient and effective?

7. What is the level of post-inspection and is it appropriate?

8. Does PPL coordinate WRAP with other weatherization programs?

9. What are the energy savings?

4


10. What is the cost-effectiveness of the various agencies?

11. Does PPL provide adequate support and training for contractors?

5

Evaluation Design

1. Background Research

2. Review of Specifications and Procedures

3. Contractor Survey

4. Baseload Observations

5. Full Cost Observations and Inspections

6

Evaluation Design

6. Customer Survey

7. Usage Impacts

8. Payment Impacts

7

Background Research• Goal – develop a complete understanding of

the WRAP procedures and implementation.• Activities

– Interview WRAP managers and staff– Review program documentation– Review program statistics

• Outputs – Understanding for research foundation– Documentation

8

Background Research

• Key Findings– PPL has five managers who oversee WRAP

and other low-income programs in their geographical area.

• Advantage: ability to focus on needs in a particular geographic area

• Disadvantage: many responsibilities – difficult to oversee work of contractors

– PPL spends much effort to come within 4% of expenditure goal (based on PUC requirement).

9

Background Research

• Key Findings– Affordability customers are prioritized.– Otherwise, jobs sent to contractors on a first

come first served basis. – Program coordination barriers

• Long waiting lists for WAP

• Long waiting lists & requirements for gas programs

• Customers who use gas and electric may not have high enough usage to qualify for either program

10

Background Research

• Recommendations– Prioritize customers based on usage.– Track program coordination and provide

incentives for contractors to coordinate services with other programs.

– Continue to introduce technological improvements, such as the web-based measure reporting form.

11

Review of Specifications and Procedures

• Goal – Determine potential effectiveness of measure selection, measure installation, and energy education

• Activities – Assess procedures and forms:– Education specifications– Education forms– Written technical procedures and manuals– Measure installation rates

12


• Outputs– Recommendations for modifications to:

• Education procedures

• Education forms

• Measure selection guidelines

• Procedures manual

13


• Education Findings– All participants receive at least one on-site

education visit– Follow-up education is provided at the time of

the inspection or by phone– Remedial education provided to customers

whose usage increases by at least 10% six months after service delivery

14

Review of Specifications and Procedures• Education Recommendations

– Education should be provided at the time of the audit. The homeowner should be present.

– Customer profile should collect information on potentially large opportunities for saving – use of dehumidifiers, use of second refrigerators, lights/appliances left on at all times.

– Action form should prioritize actions by the potential for energy saving in the individual home. Should list top 3-5 actions with estimated $ savings.

15

Review of Specifications and Procedures• Technical Findings and Recommendations*

– Review cost-effectiveness calculations in audit decision trees to reflect current estimates of costs and savings.

• Refrigerators• CFLs

– Water heater wraps and pipe insulation may be more cost effective than water heater replacement.

16*Blasnik & Associates.


• Technical Findings and Recommendations– Duct sealing in basements should be focused on

safety and comfort.– Blower door guided air sealing: investigate why

only done in 60% of full cost jobs.– WRAP standards and field guide: more concise

program field guide with separate specifications for specific areas may be useful.

17

Contractor Survey• Goal - Assess contractor compliance with program

procedures, and assess problems in program administration.

• Activities– Develop contractor survey instrument

– Determine survey sample

– Send survey to contractors

– Review completed surveys and contact respondents for additional information/clarifications

18

Contractor Survey• Outputs - Understanding of:

• Contractor background and experience

• Support and training provided to contractors

• Usefulness of program forms

• Program implementation procedures

• Joint service delivery with WAP

• Health and safety problems found in homes

• Contractors’ quality control

• Inspection issues

19

Contractor Survey

• Outputs - Recommendations related to:

– Program procedures

– Contractor training and support

– Inspection procedures

20

Contractor Survey

21

# of Contractors

Complete 16

No Response 2

Contractor Survey

22

Staff Training # of Contractors

Observing other service delivery staff 12

Being observed while delivering services

10

Classroom training 8

Affordable Comfort 6

PA WX Classes 3

WX Training Center Classes 2

Testing staff 1

Contractor Survey

23

Staff Assessment # of Contractors

Field observation of WRAP jobs 11

Practical exam 7

Professional certification 7

Written exam 5

Inspection of WRAP jobs 5

Pass Wx Training Center class 1

Web training 1

Contractor Survey

24

PPL Training Ratings

# of Contractors Who Provided Each Rating Mean

Rating1-2 3 4-5

Quality 1 4 10 3.7

Focus 1 4 9 3.6

Level 1 4 9 3.8

Amount 2 7 5 3.2

Overall 1 3 10 3.6

Contractor Survey

25

Joint Delivery of WRAP Services

# Who do Joint Delivery

% of Jobs

Min Max Mean

PA WX 8 0% 100% 16%

Gas Utility 4 0% 100% 7%

County WX 2 0% 10% 1%

Contractor Survey

26

Provide Evening and Weekend WRAP Services

Evenings Weekends

Yes 8 7

No 7 8

No Answer 1 1

Contractor Survey

27

Baseload Audit Procedures

Always Sometimes Never

Describe WRAP 10 1 0

Discuss bill with customer 11 0 0

Discuss H&S with customer 10 1 0

Conduct walkthrough with customer

8 2 1

Provide measure saving estimate 6 5 0

Provide actions savings estimate 5 6 0

Contractor Survey

28

Quality Control Methods

# of Contractors % Of Jobs

Review data collection forms

11 75%

Contact customers by telephone

10 40%

On-Site Inspection 12 41%

Observation 12 32%

Contractor Survey

29

PPL Inspectors

% of Inspections

Resolution

Inspector Fixed

Action Sheet

None Needed

Invoicing mistake 7% 17% 47% 36%

Insulation 6% 0% 83% 17%

Dryer Venting 15% 39% 61% 0%

Education 11% 100% 0% 0%

Contractor Survey

30

PPL Program Ratings

# of Contractors Who Provided Each Rating Mean

Rating1-2 3 4-5

WRAP Specs 0 3 12 4.1

PPL Communication

0 3 12 4.3

Invoicing 0 4 11 4.1

Overall 0 3 12 3.9

Contractor Survey• Recommendations

– Revisit audit forms and determine whether they can be consolidated.

– Require home walkthrough on all jobs.– Formalize a process to respond to action sheets.

31

Baseload Observations• Goals - Understand how well contractors

address opportunities for baseload usage reduction and whether education is effectively provided.

• Activities– Sample design and selection– Observation protocols– Conduct observations– Review findings and synthesis

32

Baseload Observations• Outputs - Recommendations for:

– Additional contractor training– Additional quality control

33

Baseload Observations

34

Visit Introduction

# Of Observations CommentsYes No

Visit expected 10 0

WRAP explained 6 4

Usage reviewed 4 6 2 did later in the visit

H&S discussed 4 6 2 did later in the visit

Comfort discussed 3 7 2 did later in the visit


35

Home Walkthrough

#Comments

Yes No

Inspected every room 6 41 auditor did not

do the walkthrough

Systematic inspection 6 4

Discussed electric uses 8 2

Estimate costs of uses 4 6 4 did later in visit

Reinforced costs later 8 2


36

Home Walkthrough

#Comments

Yes No

Discussed actions 7 3 1 later in visit

Estimated savings 5 5 3 later in visit

Discussed willingness to take actions

7 3 1 later in visit

Obtained customer commitment

6 4 2 later in visit


37

Home Walkthrough

MinutesComments

Min Max Avg.

Length of walkthrough 13 79 41 2 without not included in average

Part on education 13 45 26


38

Refrigerator Replacement

#Comments

Yes No NA

Monitored refrigerator 6 2 2 2 new, 1 couldn’t be moved, 1 broken

Replacement 4 5 1

Explored 2-for-1 1 2 7


39

Compact Fluorescent Light Bulbs

#

Yes No NA

Discussed all inside lights 4 6

Discussed all outside lights 7 3

Installed CFLs 8 0 2

Asked if satisfied with lighting 8 0 2

Left extra bulbs for customer 0 10


40

Energy Education

#

Yes No NA

Energy education visit 10

Engaged customer 9 1

Reviewed measures 7 2 1

Analyzed electric bill 8 2

Discussed appliances 8 2


41

Visit Summary

Minutes

Min Max Mean

Length of visit 70 180 119

Rating

Excellent Good FairNot

Acceptable

Overall rating 4 2 2 2


• Recommendations– Review WRAP requirements and expectations

with contractors.– Provide education to contractors on the

importance of 2-for-1 swaps in refrigerator replacement.

– Conduct observation of baseload service delivery.

42

Full Cost Observationsand Inspections• Goals - To understand:

– How well contractors address opportunities for electric usage reduction

– Whether correct measures were selected– Extent to which energy education is provided.

• Activities– Sample design and selection– Observation protocols– Review WRAP technical protocols– Conduction observations– Review findings and synthesis

43

Full Cost Observationsand Inspections

• Outputs – Recommendations for:– Procedures– Training

44


45

Visit Introduction

# Of Observations CommentsYes No

Visit expected 6 0

One of the customers was not present for most of the visit.

WRAP explained 4 2

Usage reviewed 4 2

H&S discussed 6 0

Comfort discussed 6 0


46

Diagnostic Testing

# Of ObservationsComments

Yes No NA

Blower door testing 5 1 One of the observations was a mobile home.

Pressure diagnostics 1 4 1


47

Missed Opportunities

1 House still connected to garage and attic.

Solar hot water.

2Remove dropped sealing and install sheetrock for an air barrier.

3 None.

4Insulate entire attic. Would have been difficult, but possible.


48

OverviewVery Good

Good Fair Comments

Data collection accuracy 1 1 2

Unable to duplicate results in one. Not all forms used in another.

Measure selection 1 3

Attic sealed shut in one so work could not be inspected.

Appropriateness of measures 2 2

One hardship case and contractor told to do everything to reduce usage.


49

OverviewExceptional Good Satisfactory Comments

Effort 3 1 Hard to assess one home because customer had moved and home unoccupied. Evaluator’s tests did not match up in

another case.

Quality 1 2 1

Appropriateness 3 1

Overall rating 1 3


• Comments– They were on the right track. They were not

afraid of working hard.– What he did was correct, but he should have

done more air sealing.– He looked at all the right things and asked the

homeowner to clarify things that were not obvious.

50


• Comments– He inspected the entire home and included the

homeowner in the audit.– He did not fill out all the forms and did not do

testing.– He was very friendly and carefully explained

everything to the customer.– He did everything according to the protocol.

51


• Recommendations– Develop one set of forms that is required for all

jobs.– Provide instructions for each form on the back

of the form.– All applicable diagnostic tests should be

required at the audit visit.

52

Customer Survey

• Goals - Understand the participants’– Demographics– Reasons for participation– Understanding of the program– Actions taken to save electricity– Bill payment difficulties– Perceived program impact on electric bills– Perceived program impact on comfort– Satisfaction with the program

53

Customer Survey

• Activities – Develop the survey instrument– Test the survey instrument– Develop a sample plan– Select the survey sample– Conduct surveys– Analyze the data

54

Customer Survey

• Outputs - Recommendations for – Program procedures– Customer education– Contractor training

55

Customer Survey

56

Demographics

Does anyone in your home have a medical condition that requires the use of additional electricity?

Yes 22%

In the past 12 months, was any member of your household unemployed and looking for work?

Yes 34%

Customer Survey

57

Why did you want to receive WRAP?

Reduce electric bills 64%

Improve comfort of the home 20%

Reduce electric usage 9%

Difficult financial situation 6%

Told to enroll 3%

Received new appliances 2%

Customer Survey

58

What are the benefits of WRAP? (Unprompted)

Energy education 36%

Lower electric bills 35%

Lower electric usage 18%

Safer or more comfortable home 11%

New appliances 9%

Improvements to home 6%

Customer Survey

59

What are the benefits of WRAP? (Prompted)




New appliances 86%


Customer Survey

60

What is the most important benefit of WRAP?





New appliances 10%

Customer Survey

61

How difficult is it for you to pay your PPL bill?

Very difficult 23%

Somewhat difficult 36%

Not too difficult 25%

Not at all difficult 13%

Customer Survey

62

Were you home for the service provider’s visit?

Home for visit 93%

Home for entire visit 85%

Customer Survey

63

Actions to Save Electricity

Provider recommended actions 83%

Provider gave savings estimates 63%

Provider gave written plan of actions 64%

Provider left information 80%

Customer Survey

64

Reduced Usage Of

Lights 72%

Dishwasher 72%

Electric heat 68%

Electric hot water heater 62%

Air conditioner 56%

Electric dryer 55%

Dehumidifier 39%

Customer Survey

65

Actions Taken (Unprompted)

Use CFLs 37%

Turn off lights not in use 18%

Keep doors/windows closed 11%

Purchase energy efficient appliances 8%

Use cold water for clothes washing 6%

Add insulation, air sealing, other measures 6%

Turn off television 2%

Turn off computer 2%

Wash only full loads 2%

Reduce hot water usage 2%

Customer Survey

• Other Actions – Clean/replace ac filters– Heat fewer rooms– Use less electric heat– Stop using an appliance– Reduce length of showers/ reduce baths– Raise refrigerator temperature– Reduce use of dishwasher

66

Customer Survey

67

Change in Comfort

Winter Summer

Improved 40% 32%

Worsened 1% 1%

No Change 58% 67%

Customer Survey

68

Program Satisfaction

Very satisfied 71%

Somewhat satisfied 22%

Somewhat dissatisfied 4%

Very dissatisfied 2%

Usage Impacts1

• Goal - estimate the actual impact of the program on customer electric usage

• Activities– Obtain program measure data

– Obtain electric usage data

– Obtain weather data

– Weather normalize the data

– Compare change for treatment and comparison groups

69Usage impact analysis done by Blasnik and Associates.

Usage Impacts

• Outputs– Estimate of the impact of the program on

energy usage– Estimate of the impact of particular program

measures– Estimate of the effectiveness of different

providers– Estimate of cost effectiveness of service

delivery

70

Usage Impacts

71

Major Measure Installation Rates

BaseloadLow Cost

Full Cost

Refrigerator replacement 50% 62% 44%

Water heater replacement 1% 59% 10%

Air conditioner replacement 18% 36% 13%

Attic insulation 0% 1% 26%

Other insulation 0% 1% 8%

Air sealing with blower door 0% 1% 15%

HVAC work 0% 0% 11%

Usage Impacts

72

Electric Savings Results

Usage Savings

# Pre Post Gross Net Net %

Baseload 659 9,661 9,394 267 836 8.7%

Low Cost 112 10,869 10,633 236 500 4.6%

Full Cost 1,019 17,912 17,129 783 1,767 9.9%

Usage Impacts

73

Measure Savings Results

Baseload Program

Savings Cost $/kWh

Refrigerator replacement 777 $662 $0.85

Air conditioner replacement 172 $546 $3.17

Usage Impacts

74

Measure Savings Results

Full Cost Program

Savings Cost $/kWh

Refrigerator replacement 532 $606 $1.14

Attic insulation 766 $882 $1.15

Other insulation 887 $999 $1.13

Windows & doors 457 $1,206 $2.64

Air sealing w/Blower door 378 $288 $0.76

Usage Impacts

75

A (75)

B (102)

C (27)

D (108)

E (22)

F (56)

G (95)

H(107)

I (36)

-1000 0 1000 2000 3000 4000Net Savings (mean ± 90% confidence intervals)

Net Savings by ProviderBaseload Program

Usage Impacts

76

Net Savings by ProviderFull Cost Program

A (166)

B (129)

C (281)

D(99)

E (58)

F (59)

G (122)

H (73)

0 1000 2000 3000 4000Net Savings (mean ± 90% confidence intervals)

Usage Impacts

• Recommendations– Reassess air conditioner replacement targeting

strategy and water heater replacement.– Refrigerator replacement, insulation, and

blower door guided air sealing should be pursued and perhaps expanded.

77

Payment Impacts

• Goal - To estimate the impact of the program on customer bills and payments.

• Activities– Obtain customer bill and payment data– Add up bills and payments in the pre and post

period– Compare change for treatment and comparison

groups

78

Payment Impacts

• Outputs – Estimate of the impact of the program on bills and payments.

79

Payment Impacts

80

Usage Savings

Pre Post Gross Net Net %

Total Bill $1,214 $1,194 -$21 -$118 -9.7%

Total Payments $1,124 $1,179 $54 -$58 -5.2%

Bill Coverage Rate 93% 100% 8% 12% 12.9%There were 1,873 customers in the treatment group and 1,228 customers in the comparison group.

Conclusions

• Several types of evaluation activities

• Each research activity brings a different set of information

• Do they tell the same story?

• Synthesis

• Recommendations that can be implemented

81

Comprehensive Usage Reduction Program Evaluation

Documents

Transcript of Comprehensive Usage Reduction Program Evaluation