CPUC Meeting on 2010 – 2012 Evaluation WO033 Custom Impact Evaluation Results May 2, 2014.

CPUC Meeting on 2010 – 2012 Evaluation

WO033 Custom Impact Evaluation Results

May 2, 2014

2

TOPICS

WO033 Custom Impact Overview

WO033 Custom Impact Elements

Evaluation Results

• Gross Impact Findings

• Net Impact Findings

• Lower Rigor Assessment (LRA) Findings

Selected WO033 Findings and Recommendations

Q&A/Discussion

Emphasis on new sampling and evaluation approaches, tailored scope allowing for measure- and program-level results, and new recommendations

WO033 Slides - Stakeholders Mtg 5/2/2014

3

WO033 CUSTOM IMPACT OVERVIEW

• Evaluation of Custom (non-deemed measures) across multiple programs and the four major California IOUs Core Calculated Programs Third Party Programs Local Government Partnerships and Universities / Colleges And More!! Over 100 programs within the WO033 scope

• Why is Custom Impact Evaluation Important? Custom measures account for 19% of electric [1,754 GWh] and

65% of statewide natural gas savings claims [134 Gigatherms] History of low realization rates for savings claims

Not Just Another Custom Impact Evaluation


4

WO033 CUSTOM IMPACT ELEMENTS

• What’s New? Two Periods, BD and AD, based on D.11-07-030 Large Gross Impact Sample Expanded Net Impact Sample Lower Rigor Assessment for Program-Level Impact

Feedback Interim Report for Early Feedback Stand-Alone Net-to-Gross Report (forthcoming) New Findings and Recommendations

These evaluation elements support program-level results for “programs of interest” and some additional measure-specific results

Not Just Another Custom Impact Evaluation


5

WO033 CUSTOM IMPACT SAMPLE SIZES

Accomplishments

(1) 429 gross impact points across five IOU-fuel analysis domains,

(2) telephone survey supporting net to gross (NTG) estimation and industry standard practice (ISP) determinations for a total of 1,388 projects, and

(3) a total of 536 engineering reviews supporting LRA assessment

IOU/Fuel Sampling Domain

Number of Completes

Gross Impact (M&V)* NTG LRA

PG&E Electric 112 558 155

PG&E Gas 75 230 97

SCE Electric 100 367 139

SDG&E Electric 73 125 73

SDG&E/SCG Gas 69 108 72

Total 429 1,388 536

* The 429 M&V sample points account for 495 electric and gas fuel-differentiated results. This is because some sample points included claims for both electric and gas savings.


6

GROSS IMPACT FINDINGS Project Lifecycle Gross Realization Rates by Fuel Domain and Energy Metric (kWh, kW, and Therms)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

SDGE ElectricSCG andSDG&E Gas*

SCE ElectricPG&E GasPG&E Electric

kWh kW Therms

1.6

n=139n=91 n=100

n=118

n=88

n=77

n=59

n=94

n=84


* Lower value excludes extreme points.

7

PROJECT LIFECYCLE GROSS REALIZATION RATES BY SAMPLE DOMAIN AND ENERGY METRIC

Energy MetricSample Size (n)

Mean Gross Realization

Rate

Population (N)

Error Ratio90% Confidence

Interval

PG&E Electric

kWh* 139 0.59 6,994 1.27 0.49 - 0.70

kW 118 0.46 6,248 1.67 0.35 - 0.58

PG&E Gas

Therms* 91 0.64 1,270 0.58 0.58 - 0.70

SCE Electric

kWh* 100 0.61 3,052 1.03 0.51 - 0.71

kW 94 0.57 2,748 1.08 0.47 - 0.67

SDGE Electric

kWh* 77 0.55 1,469 1.08 0.44 - 0.66

kW 59 0.78 790 2.20 0.43 - 1.14

SCG and SDG&E Gas

Therms, all points 88 1.45 1,077 4.82 0.28 - 2.62

Therms, less extreme points

84 0.63 1,077 0.84 0.54 - 0.73

WO033 Slides – Stakeholders Mtg 5/2/2014

Note: First year savings are also reported in Chapter 5.

8

LIFECYCLE EX-ANTE AND EX-POST ELECTRIC SAVINGS (KWH) FOR SAMPLED PROJECTS


9

LIFECYCLE EX-ANTE AND EX-POST GAS SAVINGS (THERMS) FOR SAMPLED PROJECTS


10

GROSS IMPACT FINDINGS (CONT’D)

Program-specific and Measure–specific Findings

• GRR results for program and measure groups with substantial sample sizes examined in detail

• Core, third party and new construction programs generally have above average GRR results

• While quite sensitive to outliers, lower GRR results for compressed air, HVAC and HVAC controls measure groups

• Higher GRR results for HVAC controls (PG&E

Gas), HVAC (SCE), steam traps (SCG/SDGE)


11

GROSS IMPACT FINDINGS (CONT’D)Reasons for Discrepancies in Realization Rates

Three principal reasons observed operating conditions baseline specification the IOUs calculation methods

• Explain differences for 44%, 17% and 19% of the records studied • Impact on savings

observed operating conditions -17% (electric), -14% (gas) baseline specification -16% (electric), -14% (gas) the IOUs calculation methods -2% (electric), -1% (gas)

• Other reasons for differences incorrect equipment specifications ineligible measures incorrect measure counts tracking database discrepancy


12

FREQUENCY OF DISCREPANCY FACTORS


44%

19%17%

6%

3% 3%1%

4% 2%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Operating Conditions

Calculation Method

Inappropriate Baseline

Ineligible Measure

Equipment Specifications

Measure Count Tracking Data Discrepancy

Other No Significant Discrepancy

13

EFFECT OF DOMINANT DISCREPANCY FACTORS ON TOTAL EX-ANTE KWH SAVINGS CLAIMS


14

GROSS IMPACT FINDINGS (CONT’D)Non-Residential New Construction

• 43 sites in the NRNC measure group (25 EnergyPro)• GRR: 81% (kWh), 72% (kW) and 51% (therms)

• Primary discrepancies Operating conditions Calculation methods Inappropriate baselines

• EnergyPro-related issues All whole building sites used T-24 schedules instead of the as-

designed scheduled to estimate the ex-ante savings Ex ante energy models were not created or updated to match the

physical as-built conditions of the building Operating under sub-optimal conditions, e.g., low occupancy

• Non EnergyPro-related issues Use of inappropriate baseline on data center projects Estimated I.T. loads on data centers were higher than observed


15

GROSS IMPACT FINDINGS (CONT’D)Monitoring-Based Commissioning

• 10 sites in the MBCx measure group GRR: 58% (kWh), 108% (kW) and 34% (therms)

• Primary discrepancies Validity of ex ante regression model Retrofit measures implemented during and after MBCx Benchmarking of project sites Reliability of energy meters and flawed metered data Changes incompatible with the building equipment capability Negative energy savings that may not be due to MBCx

measures Adjustments to energy usage baseline

WO033 Slides - Stakeholder's Mtg 5/2/2014

16

NET IMPACT FINDINGSNet to Gross Completes

• Mix of CATI and In Depth/Professional Interviews • Weighted NTGRs by program or program groupings

0

0.2

0.4

0.6

0.8

1.0

SDGE ElectricSCG andSDG&E Gas

SCE ElectricPG&E GasPG&E Electric

kWh Therms

n=558n=230

n=367 n=108 n=125


17

NET IMPACT FINDINGSNet to Gross Results

Results

Electric NTGRs Gas NTGRs

Statewide PGE SCE SDG&E Statewide PGE SDG&E/SCG

Weighted NTGR 0.47 0.46 0.49 0.45 0.53 0.56 0.50

90 Percent Confidence

Interval 0.46 - 0.48

0.45 -

0.48 0.47 - 0.50 0.43 - 0.48 0.50 - 0.57 0.53 - 0.58

0.42 -

0.57

Relative Precision 0.03 0.04 0.04 0.06 0.07 0.05 0.15

n NTGR Completes 1,050 558 367 125 338 230 108

N Sampling Units 11,515 6,994 3,052 1,469 2,347 1,270 1,077

Error ratio (ER) 0.52 0.59 0.44 0.43 0.79 0.46 0.99


18

NET IMPACT FINDINGSNet to Gross Findings

• Significant levels of free ridership have persisted Statewide NTGRs of 0.47 (electric) and 0.53 (gas) Similar in magnitude to NTGRs from prior cycles (Significant improvement in PG&E gas NTGR from

PY2006-2008)• Little evidence of actions taken to reduce free ridership via

changes in program design or implementation procedures • Factors causing non-zero levels of free ridership:

Size and sophistication of customers. Custom program design

• Despite these challenges, many strategies available to reduce free ridership

Many recommended in previous cycles


19

OVERALL NET REALIZATION RATES

• PG&E kWh – 0.27, kW – 0.22, Therms – 0.36 Evaluated as % of claimed – kWh – 44%, kW – 34%,

Therms – 59%

• SCE kWh – 0.30, kW – 0.28 Evaluated as % of claimed – kWh – 49%, kW – 46%

• SDG&E/SCG kWh – 0.25, kW – 0.36, Therms – 0.72 Evaluated as % of claimed – kWh – 50%, kW – 53%,

Therms – 124%*

* 54% excluding extreme points.


20

LRA Overview

LOWER RIGOR ASSESSMENT FINDINGS

• Uses a desk review process similar to EAR

• Provides early feedback based on qualitative information

• Findings by program/program group, fuel domain, and IOU

• Expanded LRA form in AD period

• Results supported by a minimum of 8 assessments/category

• Low-frequency issues (n<100) use “GNP” denominator, see next slide

• Tables highlight strengths of project documentation related patterns, trends areas for improvement

• LRA Score in Final Report – program-level LRA performance


21

LOWER RIGOR ASSESSMENT FINDINGS (CONT’D)


Issue Assessed Assessment Results (n)

Assessment Results (%)

Good Neutral Poor (OD) Poor (GNP)

Project Documentation and Specification

IOU application documentation complete and accurate 513 39% 33% 26% 28%

IOU tracking data complete and accurate 520 38% 42% 20% 20%

Project utilized pre-installation M&V 390 43% 27% 22% 30%

Appropriate baseline 507 64% 14% 21% 22%

Early replacement claim: Valid RUL / EUL approach used 249 48% 12% 18% 40%

Appropriate Calculation Method

Appropriate impact calculation method 492 43% 34% 21% 23%

All relevant inputs considered 458 63% 16% 18% 21%

Adequate values for all inputs 470 37% 40% 21% 23%

Appropriate HVAC interactive effects calculation method 67 55% 12% 4% 33%

Appropriate non-HVAC interactive effects calculation method 77 62% 17% 3% 21%

Project utilized post-installation M&V 505 33% 34% 30% 32%

Compliance with Program Rules Measures are IOU program eligible 520 84% 13% 3% 3%

Measures exceed code or industry standard practice 473 75% 12% 11% 13%

Multiple IOU fuel impacts properly accounted for (includes fuel switching and cogeneration)

44 43% 5% 4% 52%

If applicable, fuel switching supported with three prong test 16 31% 0% 2% 69%

Non-IOU fuel and ancillary impacts of project properly accounted for (cogen/waste heat recovery/ refinery gas, etc.)

94 27% 0% 13% 73%

Customer installation meets all program rules 518 75% 12% 13% 13%

Final Report Table 7-1: LRA Complete Portfolio Sample

Issue areas with fewer than 100 responses included only LRAs with valid responses in the denominator as shown in “Assessment Results” column, otherwise the count of all LRAs (536).

22

LOWER RIGOR ASSESSMENT FINDINGS (CONT’D)LRA Results Wrap-up

Issues with high rate of deficiencies (percent “poor” shown in parentheses):

• Project documentation (26%)• Pre- and post-retrofit M&V (22% and 30%)• Calculation methods and inputs, specifically HVAC interactive

effects (33%)

Issues with high rates of poor assessments within lower-frequency issue areas

Accounting for non-IOU fuel and ancillary impacts (73%)Accounting for multiple IOU fuel impacts (52%)Fuel switching supported with 3-prong test (69%)

Lower rigor assessment qualitative outcomes are not necessarily indicative of gross impact evaluation results (LRA Score vs GRR: R2=0.04 for kWh)


23

LOWER RIGOR ASSESSMENT FINDINGS (CONT’D)Final Report Table 7-3: IOU Fuel Domain Program Performance


Issue Assessed IOU Fuel Domain

PG&E Electric

PG&E Gas SCE SCG and

SDG&E Gas SDG&E Electric All LRA Average

Number of Assessments 172 97 139 72 73 536

Number of “above average” issues 5 6 3 3 1 –

Number of “below average” issues 1 1 1 6 11 –

Overall Assessment Score 0.73 0.93 0.33 -0.50 -1.83 -0.50

Project Documentation and Specification IOU application documentation complete and accurate 26% 23% 30% 17% 36% 26%

IOU tracking data complete and accurate 13% 22% 20% 24% 25% 20%

Project utilized pre-installation M&V 20% 15% 24% 25% 30% 22%

Appropriate baseline 14% 15% 22% 24% 37% 21%

Early replacement claim: valid RUL / EUL approach used 20% 13% 16% 17% 29% 18%

Appropriate Calculation Method

Appropriate impact calculation method 18% 14% 22% 21% 33% 21%

All relevant inputs considered 12% 19% 20% 21% 25% 18%

Adequate values for all inputs 20% 21% 21% 22% 18% 21%

Appropriate HVAC interactive effects calculation method 18% 11% 56% Small Sample 33% 33%

Appropriate non-HVAC interactive effects calculation method

29% 12% 17% 36% 0% 21%

Project utilized post-installation M&V 23% 26% 33% 38% 40% 30%

Compliance with Program Rules Measures are IOU program eligible 3% 7% 1% 1% 3% 3%

Measures exceed code or industry standard practice 12% 11% 6% 22% 10% 11%

Multiple IOU fuel impacts properly accounted for 60% 42% Small Sample 38% 70% 52%

If applicable, fuel switching supported with three prong test Small Sample 57% Zero Sample Small Sample 100% 69% Non-IOU fuel and ancillary impacts accounted for (cogen/waste heat recovery/ refinery gas, etc.)

84% 56% 88% 64% 71% 73%

Customer installation meets program rules 10% 14% 9% 17% 19% 13%

24

WO033 REPORT RECOMMENDATIONS

• Project Savings Estimation Improve implementation and QC of project operating

conditions, ex-ante baselines, calculation methods Improve accuracy of savings in IOU tracking databases

based on post-installation inspections and M&V Improve engineering and simulation modeling for novel

measures and projects with limited access to data• Sufficient Quality Control for Documentation and Tracking

Data Increase consistency between project files and tracking

data Minimize miscommunication of project claims. Provide additional scrutiny and QC

o Accurate, comprehensive project documentationo Internally reviewedo Consistent with tracking system entries

Provide electronic access to complete documentation


25

WO033 REPORT RECOMMENDATIONS (CONT’D)

• IOU Project Verification Perform pre and post-retrofit verification and M&V

more frequently, document results completely Goal to accurately identify

o pre-existing equipmento ex-ante operating conditionso baseline considerations, and other issues

• LRA Evaluation Reduce documentation inconsistencies Improve impact calculation methods Comply with program rules or justify exceptions Conduct LRA-type reviews to enhance project

reviews


26


• Nonresidential New Construction Use as-built schedule to update estimates of ex-ante

savings and use “non-compliance” mode calculations for savings

Submit a Title-24 acceptance test report, perform site visits to verify key measures and revise energy model to physical “as-built” conditions

Provide technical outreach assistance to all program participants

Thoroughly review and make appropriate use of the data center baseline document

Conduct an ISP study focused on changes in load over time for various types and sizes of data centers


27


• MBCX

Develop guidelines for regression modeling, review early stage projects

ensure conformance, applicability, exceptions Keep centralized records of all building operation changes,

especially for higher education campuses For project benchmarking, use historical data for buildings with

similar characteristics, test modeling with independent variables Use meters that conform to the adopted metering guidelines Verify that MBCx recommendations address the control system.

Recommended changes must be compatible with existing system

Retain records of each measure implemented, along with post-implementation functional performance tests

List specific activities done, verify work was done correctly Perform more extensive and thorough data collection, analysis,

model development and validation of the engineering models


28


• Program Influence/NTG Adopt procedures to screen for and increase

efficiency levels for high likelihood free ridership projects. Advise customers to

o move to a higher level of efficiencyo undertake a bundled retrofit to ensure

deeper savingso move to another EE project under

consideration Carefully review the list of qualifying measures;

eliminate eligibility for standard practice measures Make changes to the incentive design. Tier

incentives by technology class. Consider adopting a payback floor.


THANK YOU

www.itron.com

CONSULTING & ANALYSIS GROUP

1111 Broadway

Oakland, CA

CPUC Meeting on 2010 – 2012 Evaluation WO033 Custom Impact Evaluation Results May 2, 2014.

Documents

Transcript of CPUC Meeting on 2010 – 2012 Evaluation WO033 Custom Impact Evaluation Results May 2, 2014.