Comparison of Pooled and Household-Level

Usage Impact AnalysisJackie Berger

Ferit Ucar

IEPEC Conference – August 14, 2013

Presentation Outline

• Motivation

• Billing Analysis

• Usage Impact Models

• Model Results

• Summary and Next Steps

2

MOTIVATION

3

Usage Impacts

• Were expected energy savings results obtained?

• Are the treatments cost-effective?

• Should measure selection procedures be revised?

• Should installation procedures be reviewed?

• Should contractors be re-trained?

4

Analysis Method

• Goal: develop most accurate estimate of program savings.

• Weigh costs and benefits of various approaches to measurement.

• Consider possible causes of mis-measurement or bias.

5

What Are You Measuring?

6

Approach Measures Issues

Engineering Estimate

Expected usage change based on measures alone

•Assumptions•Installation quality•Other usage changes

Usage2 - Usage1 Actual change in usage•Weather •Other factors

Weather Norm Usage2 - Usage1

Change in usage if both periods had average weather

•Other factors

Weather Norm Usage2 - Usage1

w/Comp. Group

Other factors held constant (prices, economy, market information, etc.)

•Best estimate of program impact

Analysis Approaches

7

ApproachCost Accuracy Attrition

Reasons for Exclusion

Engineering Estimate

$ **** None

Engineering Estimate with Retrofit Data

$$ * ***Retrofit Data

Missing

Billing Analysis

$$$ *** **Usage Data Missing or Inadequate

Metering $$$$ **** * Cost

Billing Analysis

8

Data Requirements

9

Core Data Required Supplemental Data

Energy billing data

Read date, real or estimated, usage, units

Energy efficiency measures

Measure-specific impacts

Service delivery date

Divides period into pre- and post-treatment

Service delivery providers

Provider-specific impacts

Weather data

Local weather station, daily temp for pre and post period and longer normalization period

Housing unit characteristics

Relation between housing /household characteristics and savings

Household characteristics

Challenges

10

Concern

Data AttritionSavings do not represent treated population

Sample SizeLow precision for savings estimatesCannot estimate for sub-groups

Comparison Group

Need to control for exogenous factorsNot able to do random assignmentDifficult to find comparable population•Later program participants•Earlier program participants•Comparable households

Treatment and ComparisonGroup Example

11

Post-Treatment Period

PreYr 2 – Quasi Post

2010

Post Yr 2 – Quasi PostSERVICE DELIVERY DATE

SERVICE DELIVERY DATE

2011 2012

SERVICE DELIVERY DATE

Pre-Treatment Period

Post Yr 1 – Quasi Pre

Pre Yr 1 – Quasi Pre

Treatment Group

ComparisonTreated Year BeforeComparisonTreated Year After

Usage Impact Models

12

House-by-House

13

Pooled Analysis

14

Fit = average daily usage during the pre- and post-treatment periodsHit = average daily base 60 HDDsPOSTt = a dummy variable that is 0 in the pre-period and 1 in the post-period εit = estimation error term

• PRE USAGE• αi = average daily baseload usage in pre-treatment period.• β1 = average daily usage per HDD in the pre-treatment period.

• POST USAGE• αi + β2= average daily baseload usage in the post-treatment period. • β1 + β3= average daily usage per HDD in the post-treatment period.

• SAVINGS• β2 = average daily baseload savings• β3 = heating usage savings per HDD.

Fit= αi+ β1* Hit+ β2*POSTt+ β3*POSTt*Hit+ εit

Advantages

15

House-by-House Pooled

Detailed attrition analysisUtilizes all billing data

Post-analysis of usage and savings Direct estimate of savings is furnishedAnalysis of high- and low-saving homes

Regression models can be used to estimate savings by measure

Exogenous factors can be included in the model

Relationship between usage and housing/ household characteristics can be explored

Disadvantages

16

House-by-House Pooled

Less robust when energy use response to degree days varies within household

Inclusion of many parameters can make final results difficult to interpret

Requires close to a full year of pre- and post-treatment data

Alternative functional form may be required

Substantial attrition can bias the analysis

Limited ability to furnish information on savings distribution and conduct exploratory analysis

When to Use

17

House-by-House Pooled

Availability of close to one year of pre-and post-treatment usage data for significant % of treatment and comparison

Limited data availability and concern for attrition bias associated with excluding homes

Data available on treatment, home, or households, that can be used to assess factors related to higher or lower savings

Supplemental data not availableStudy sponsors not interested in supplemental analysis

Model Results

 

18

Program 1 ResultsGas Heating Jobs

19

Model Obs.Pre-Use

Post-Use

Savings

ccf %

Not Normalized 1,166 1,060 990 70 (±11) 6.6%

House-by-House 1,166 1,052 991 61 (±10) 5.8%

Pooled Regression 1,166 1,030 964 66 (±10) 6.4%

Pooled-Month Dummy 1,166 1,084 1,020 64 (±10) 5.9%

Pooled-all obs. 1,439 1,031 966 65 (±9) 6.3%

Program 2 ResultsGas Heating Jobs

20

Model Obs.Pre-Use

Post-Use

Savings

ccf %

Not Normalized 1,211 1,042 831 210 (±12) 20.2%

House-by-House 1,211 1,025 959 67 (±10) 6.5%

Pooled Regression 1,211 999 936 63 (±9) 6.3%

Pooled-Month Dummy 1,211 1,044 976 68 (±10) 6.5%

Pooled-all obs. 1,665 1,002 933 69 (±8) 6.9%

Program 1 ResultsElectric Baseload Jobs

21

Model Obs.Pre-Use

Post-Use

Savings

kWh %

Not Normalized 4,055 11,153 10,792 361 (±73) 3.2%

House-by-House 4,055 11,370 10,147 1,223 (±78) 10.8%

Pooled Regression 4,055 10,624 9,735 889 (±55) 8.4%

Pooled-Month Dummy 4,055 10,798 9,957 841 (±56) 7.8%

Pooled-All Obs. 5,375 10,728 9,893 835 (±53) 7.8%

Pooled-Month-All 5,375 11,190 10,425 765 (±55) 6.8%

Program 2 ResultsElectric Baseload Jobs

22

Model Obs.Pre-Use

Post-Use

Savings

kWh %

Not Normalized 2,440 11,022 9,765 1,257 (±93) 11.4%

House-by-House 2,440 10,758 10,148 610 (±99) 5.7%

Pooled Regression 2,440 10,139 9,501 638  (±69) 6.3%

Pooled-Month Dummy 2,440 9,779 9,123 656  (±82) 6.7%

Pooled-All Obs. 4,654 10,287 9,726 561  (±56) 5.5%

Pooled-Month-All 4,654 9,853 9,277 575  (±66) 5.8%

Program 1 ResultsElectric Heating Jobs

23

Model Obs.Pre-Use

Post-Use

Savings

kWh %

Not Normalized 144 17,84617,77

967 (±541) 0.4%

House-by-House 144 19,66218,53

41,128 (±503) 5.7%

Pooled Regression 144 17,94017,08

4857 (±559) 4.8%

Pooled-Month Dummy 144 19,73818,82

6912 (±586) 4.6%

Pooled-All Obs. 220 17,83016,99

2838 (±491) 4.7%

Pooled-Month-All 220 20,84620,02

8818 (±515) 3.9 %

Program 2 ResultsElectric Heating Jobs

24

Model Obs.Pre-Use

Post-Use

Savings

kWh %

Not Normalized 134 18,103 14,298 3,805 (±646) 21.0%

House-by-House 134 19,402 17,899 1,503 (±665) 7.7%

Pooled Regression 134 17,020 15,505 1,515 (±543) 8.9%

Pooled-Month Dummy 134 17,177 15,614 1,562 (±647) 9.1%

Pooled-All Obs. 282 16,884 15,263 1,621 (±391) 9.6%

Pooled-Month-All 282 18,193 16,374 1,819 (±457) 10.1%

Electric Baseload

Pre-Treatment Usage Obs.Pre-Use

Post-Use

Savings

kWh %

< 8,000 kWh 703 6,862 6,568 294 4.3%

8,000 – 12,000 kWh 1,972 9,720 8,829 891 9.2%

> 12,000 kWh 1,380 16,024 13,852 2,172 13.6%

Program 1 ResultsHousehold Characteristics

25

Owner Obs.Pre-Use

Post-Use

Savings

kWh %

Owner 2,263 11,433 10,155 1,278 11.2%Renter 1,792 11,291 10,137 1,154 10.2%

Electric Baseload

Supplemental Heat Obs.Pre-Use

Post-Use

Savings

kWh %

Supplemental Heat 2,054 12,293 10,812 1,481 12.0%

No Supp Heat 2,001 10,423 9,464 959 9.2%

Program 1 ResultsHousehold Characteristics

26

Level of Service Obs.Pre-Use

Post-Use

Savings

kWh %

Basic 3,533 11,291 10,161 1,130 10.0%Major 522 11,906 10,051 1,855 15.6%

Major measures include refrigerators, air conditioner, and water heater replacements.

Electric Baseload

Measures Obs.Pre-Use

Post-Use

Savings

kWh %

Air Conditioner 78 11,363 9,626 1,737 15.3%

No Air Conditioner 3,977 11,370 10,157 1,213 10.7%

Refrigerator 420 11,211 9,376 1,386 12.4%

No Refrigerator 3,635 11,388 10,235 1,153 10.1%

AC/Refrigerator 27 11,432 9,232 2,200 19.2%

AC/ No Refrigerator 51 11,327 9,834 1,493 13.2%

No AC/ Refrigerator 393 11,196 9,386 1,810 16.2%

No AC/ No Refrigerator 3,584 11,389 10,241 1,148 10.1%

Program 1 ResultsHousehold Characteristics

27

Summary and Next Steps

28

Summary

• Overall savings results fairly consistent

• Differences between models rarely statistically significant

• Gas usage results were more consistent

• Electric baseload varied most

29

Conclusions

• Sources and potential biases caused by large data attrition should be explored.

• When additional analysis is desired for many subgroups and data attrition is low, house-by-house may be favored.

• When data attrition is high and only overall usage results are desired, the pooled regression may be preferred.

30

Next Steps

• Additional exploration of differences.

• Explore deletion of various types and numbers of observations from house by house.

• Compare results with different levels of attrition.

• Test different functional forms for the pooled model.

31