Electric / Gas / Water MAESTRO Evaluation Showcase July 26-27, 2006 Project Manager: Pierre Landry,...

Electric / Gas / Water

MAESTRO Evaluation ShowcaseJuly 26-27, 2006

Project Manager: Pierre Landry, SCELead Consultants: Mike Rufo, Itron; Keith Rothenburg,

Southern Exposure Engineering; KEMA Inc.

What a Realization: An Ex Post Impact Evaluation of a

Performance-Based Program(2002/2003 SPC Evaluation)

Knowledge to Shape Your Future

2

Overview of Presentation

• Evaluation Context and Background• Approach• Results and Key Findings• Recommendations


3

Context: Custom, Bidding, and SPC Type Programs

• History and types– custom rebates, bidding, SPC

• Goals– resource acquisition and market transformation

• Common features– focus on custom efficiency measures in large C&I– encouragement of comprehensive projects – inclusion of technical engineering review – requirements for proof of project installation


4

Common Issues for Large C&I Programs

• Uncertainty in savings estimates • Risk of gaming and fraud • Costs of measurement and verification • Keeping application requirements manageable yet

effective• Distributing funds equitably• Minimizing free riders/maximizing net impacts• Supporting the efficiency services market


5

Background on 2002-2003 CA SPC

• 3 IOUs (PG&E, SCE, SDG&E)• Targeted at Large C&I• Incentives by end use

– Lighting ($0.05), HVACR ($0.14), Process/Other ($0.08)

• Calculated or Measured Savings Path– Determined by utility– 90+% calculated

• All projects require Site Installation Report


6

Why An Ex Post Evaluation of “Performance Contract” Program?

• PY2002 & PY2003 first years SPC had independent ex post impact evaluation

• Originally (’98/’99), all SPC projects required in-program M&V– Initially, 2 years of M&V, then 1 year

• MT policy environment in CA (1998-2000)– Virtually no attention to ex post impact evaluation

• M&V dropped as requirement in 2000– Utilities retained right to require– Most projects done under calculated savings path

• Shift back to resource acquisition focus with energy crisis and 2006-2008 program admin decision


7

• Objectives– Re-establish process for

site-specific ex post impact evaluation

– Implement process for representative sample

– Estimate program realization rate

– Estimate net-to-gross

– Recommendations for improvement

• Constraints– Total budget vs. program

size, project complexity, # of utilities

– Per site budgets

• Limited sub-metering

– Took 2 years for most of sample to install projects

– Limited pre-metering available

– Self report NTGR method

PY2002 SPC Evaluation Objectives and Constraints


8

Approach

• Develop and implement sample design (ratio estimation) • Obtain files & savings documentation • Review applications and prepare ex post analysis plans• Conduct on-site data collection (limited metering)• Develop ex post impact estimates• Prepare detailed, site-specific documentation• Carry out quality control review• Extrapolate final ex post results to the population• Estimate free ridership

– self-report method• CADMAC protocol-level battery


9

Population

• Combined 2002-2003 SPC Population (apps):

• Combined 2002-2003 SPC Population (kWh):

kWh strata Lighting HVAC/R Process/Other TotalTier 0 1 1Tier 1 & Tier 2 66 40 88 194Tier 3 157 166 433 756Total 223 206 522 951

kWh strata Lighting HVAC/R Process/Other TotalTier 0 27,179,804 27,179,804Tier 1 & Tier 2 116,940,217 70,009,716 177,655,302 364,605,235Tier 3 48,613,969 38,606,325 95,633,014 182,853,308Total 165,554,186 108,616,041 300,468,120 574,638,347


10

Sample Design

• PY2002-PY2003 SPC Sample (apps)

PY2002-PY2003 SPC Sample (kWh)

kWh strata Lighting HVAC/R Process/Other TotalTier 0 1 1Tier 1 & Tier 2 12 10 16 38Tier 3 6 8 12 26Total 18 18 29 65

kWh strata Lighting HVAC/R Process/Other TotalTier 0 27,179,804 27,179,804Tier 1 & Tier 2 26,305,148 26,659,630 36,845,550 89,810,328Tier 3 2,029,233 3,506,658 2,614,565 8,150,456Total 28,334,381 30,166,288 66,639,919 125,140,588


11

Gross Realization Rate Results

0

5,000

10,000

15,000

20,000

25,000

30,000

0 5,000 10,000 15,000 20,000 25,000 30,000

Ex Ante MWh

Ex

Pos

t M

Wh

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

10,000

0 2,000 4,000 6,000 8,000 10,000

Ex Ante MWh

Ex

Pos

t M

Wh


12

Gross Realization Rate Results

Sampling Strata Lighting HVAC/R Process/OtherTier 0 -- -- 0.38Tier 1 & 2 Combined 0.92 0.96 0.88Tier 3 0.93 0.84 0.99

0.890.83 to 0.96

Total Weighted Gross Impact Realization Rate90 Percent Confidence Interval


13

Freeridership and Net-to-Gross

Sampling Strata

Net-of-Free-Ridership

RatioTier 0 and 1 0.43Tier 2 0.49Tier 3 0.54

Total Weighted Net-of-Free-Ridership 0.4890 Percent Confidence Interval 0.42 to 0.55

Weighted Net-of-Free-Ridership Value 0.48

Adjustment to Account for Self-Report Bias + 0.10

Adjustment to Account for Spillover + 0.05

Estimated 2003 SPC Net-to-Gross Ratio = 0.63

(1 – Free Ridership) 1998 1999 2000 2001 2002 2003

Weighted 0.53* 0.51 0.41 0.65 0.45 0.59

Unweighted 0.49 0.48 0.46 0.55 0.45 0.60

DEER SPC NTGR = 0.7


14

Key Findings

• PY2002-2003 SPC more like a custom rebate than performance contract program

• Overall realization rate reasonably high but below one• Wide range in individual realization rates• Importance of influence of largest projects• Gross savings may not be as conservative as SPC

program managers intended• NTGR slightly lower than 1998-2001 period


15

Other Savings-Related Key Findings

• Wide range in the quality of applications and supporting documentation

• Need for increased verification and documentation of assumptions in project files

• Experience and expertise levels of the reviewers vary widely

• Difficulties in assessing complex industrial process projects

• Limited estimation of kW peak demand savings


16

Impact-Related Recommendations

• Consider targeted increases in the level of technical documentation

• Consider a stronger application affidavit statement regarding savings assumptions

• Further standardize the review approach and documentation requirements for recurring complex projects

• Consider providing or requiring more technical support for applicants for complex projects


17

• Improve reviewer documentation • Consider increasing:

– conservatism for calculated path savings estimates– measurement for large complex projects– incentive premium for measured projects

• Increase pre-installation measurement for very large projects with highly uncertain baseline conditions

• Consider independent review of the SPC calculator

Savings-Related Recommendations (Cont.)


18

• Increase efforts to reduce free ridership• Consider:

– Higher incentive levels for higher payback measures or emerging technologies

– Incorporating a payback floor– Bonus payments for first-time participants– Custom baselines for process improvements– Excluding projects that are obvious free riders

Free Ridership-Related Considerations


19

• Consider shifting to ex post impact evaluations from program-year to paid-year basis – or a combination

• Expand scope of ex post measurement– more measurement per site– more sites in samples – incorporate uncertainty in ex post estimates into program

realization rate confidence interval

• Integrate evaluator early into program process to enable pre-measurement where necessary

Evaluation-Related Recommendations

Electric / Gas / Water MAESTRO Evaluation Showcase July 26-27, 2006 Project Manager: Pierre Landry,...

Documents

Transcript of Electric / Gas / Water MAESTRO Evaluation Showcase July 26-27, 2006 Project Manager: Pierre Landry,...