Workplan for 2018-19 IALC Custom Impact Evaluation · Total 2,449 140 51% 55% 5% 229 95 82% 5% ....

Report Group D – D01.01

Workplan for 2018-19 Industrial, Agricultural, and Large Commercial (IALC) Custom Impact Evaluation

Submitted to California Public Utilities Commission 505 Van Ness Avenue San Francisco, CA 94102

Submitted by SBW Consulting, Inc. 2820 Northup Way, Suite 230 Bellevue, WA 98004

In association with BuildingMetrics Inc.

Energy350

Opinion Dynamics

Ridge & Associates

Tierra Resource Consultants

June, 7 2019

Workplan for 2018-19 IALC Custom Impact Evaluation

SBW Consulting, Inc. ii

Acknowledgements

We want to acknowledge the critical contributions from CPUC Energy Division staff, who

guided the development of the workplan and provided many improvements and clarifications

regarding CPUC policies and relevant practices.


SBW Consulting, Inc. iii

Table of Contents

List of Figures .............................................................................................. v

List of Tables .............................................................................................. vi

List of Acronyms and Abbreviations ............................................................ vii

Executive Summary ..................................................................................... 1

Sample Design .......................................................................................................... 1

Methodology ............................................................................................................. 2

Schedule ................................................................................................................... 2

1 Background .......................................................................................... 3

1.1 Prior Evaluations .............................................................................................. 3

1.2 Issues Identified in ESPI Performance Memos ................................................... 3

1.3 CPUC Policies and Guidance .............................................................................. 4

2 Evaluation Objectives ............................................................................ 6

3 Review of Claims Database ................................................................... 7

3.1 Data Requests ................................................................................................. 7

3.1.1 Review Project Claims .............................................................................................. 7

3.1.2 Request Supplemental Data ..................................................................................... 7

3.2 Review Reported Claims ................................................................................... 8

3.2.1 Correct Claims Based on Supplemental Data ............................................................. 8

3.2.2 Claim Year ............................................................................................................... 8

3.2.3 Classification Based on Measure Description ............................................................. 8

3.2.4 Net-to-Gross Ratios.................................................................................................. 9

3.2.5 Effective Useful Lifetimes ....................................................................................... 10

3.3 Calculate Lifetime Net Savings ........................................................................ 10

4 Sample Design .................................................................................... 12

4.1 Gross Savings Sample for 2018 ...................................................................... 12

4.1.1 Sample Frame ....................................................................................................... 12

4.1.2 Sample Domains, Size and Expected Precision ........................................................ 13

4.2 Net Savings Sample for 2018 .......................................................................... 16

4.3 Gross and Net Savings Sample for 2019 .......................................................... 16

5 Ex Post Evaluation Methods ................................................................. 17

5.1 Data Requests ............................................................................................... 17

5.1.1 Obtain Project Files ................................................................................................ 17

5.1.2 Request Missing Data, Models, or Documents ......................................................... 17

5.2 Gross Savings Evaluation ................................................................................ 18

5.2.1 Prepare Site-Specific M&V Plans ............................................................................. 18

5.2.1.1 Features of All M&V Plans ................................................................................. 19


SBW Consulting, Inc. iv

5.2.1.2 New Construction ............................................................................................. 21

5.2.1.3 Site-Specific NMEC Projects .............................................................................. 22

5.2.2 Determine Ex Post Conditions ................................................................................. 23

5.2.3 Re-Estimate Project Gross Savings ......................................................................... 25

5.3 Net Savings Evaluation ................................................................................... 27

5.3.1 Questionnaire Development ................................................................................... 27

5.3.2 Data Collection ....................................................................................................... 28

5.3.3 NTGR Analysis ........................................................................................................ 29

5.3.3.1 NTGR Reliability Analysis .................................................................................. 29

5.3.3.2 NTGR Sensitivity Analysis ................................................................................. 29

5.3.3.3 Assessing the Drivers of the NTGR .................................................................... 29

Quantitative Approach ........................................................................................................... 30

5.4 Cost, Lifetime, and Load Shape ....................................................................... 31

6 Portfolio Savings and Cost-Effectiveness .............................................. 32

6.1 Gross Savings ................................................................................................ 32

6.2 Net Savings ................................................................................................... 34

6.3 Cost-Effectiveness .......................................................................................... 36

7 Data Management and Quality Control ................................................ 37

8 Task Plan ............................................................................................ 38

8.1 Workplans and Updates .................................................................................. 38

8.2 Data Collection and Sampling Plans ................................................................ 38

8.2.1 Sample Frame ....................................................................................................... 38

8.2.2 Review of Reported Custom Claims ........................................................................ 38

8.2.3 Sample Selection ................................................................................................... 38

8.2.4 Data Requests Related to Sampled Custom Projects ................................................ 38

8.2.5 Site Data Collection ................................................................................................ 39

8.3 Gross Savings Estimates................................................................................. 39

8.4 Net Savings Estimates .................................................................................... 39

8.5 Final Reports ................................................................................................. 39

8.6 Data Documentation ...................................................................................... 40

9 Budgets and Schedule ......................................................................... 41

9.1 Evaluation Budget .......................................................................................... 41

9.2 Timeline and Milestones ................................................................................. 41

10 Management and Staffing Plan ............................................................ 43


SBW Consulting, Inc. v

List of Figures

Figure 1: Schedule for Ex Post Evalution of 2018 and 2019 Custom Projects .................................. 2

Figure 2: Timeline and Milestones .............................................................................................. 42

Figure 3: IALC Evaluation Organizational Chart ........................................................................... 43


SBW Consulting, Inc. vi

List of Tables

Table 1: Sample Design for Gross Savings Evaluation of 2018 Custom Projects .............................. 1

Table 2: Example of Keyword List ................................................................................................. 9

Table 3: 2018 IALC Sample Frame Summary .............................................................................. 13

Table 4: 2018 IALC Sample Design – Electric Frame .................................................................. 14

Table 5: 2018 IALC Sample Design – Gas Frame ......................................................................... 15

Table 6: 2018 IALC Sample Design Summarized by Program Administrator – Electric Frame ........................................................................................................................................ 15

Table 7: 2018 IALC Sample Design Summarized by Program Administrator – Gas Frame ........... 15

Table 8: IALC Evaluation Budget ................................................................................................. 41


SBW Consulting, Inc. vii

List of Acronyms and Abbreviations

Acronym/ Abbreviation

Definition

AQMD Air quality management districts

CATI Computer-assisted telephone interview

CMPA Custom measure and project archive

CPUC California Public Utilities Commission

EM&V Evaluation, measurement and verification

ER Early retirement

ESPI Efficiency savings and performance incentive

EUL Effective useful life – The average time over which an energy efficiency measure results in energy savings, including the effects of equipment failure, removal, and cessation of use.

HOPPs High opportunity projects and programs

IALC Industrial, agricultural, large commercial

IOUs Investor owned utilities

IPMVP International performance measurement and verification protocol

ISP Industry standard practice

kW Kilowatts

kWh Kilowatt-hours

M&V Measurement and verification

MMBtu Millions of British thermal units

NMEC Normalized metered energy consumption – This is a means by which savings are quantified at the meter level through a regression approach that normalizes energy use by independent variables such

as weather, production, etc.

NTGR Net-to-gross ratio

PA Program administrators

PG&E Pacific Gas and Electric

POE Preponderance of evidence

RUL Remaining useful life – The number of remaining years that an item, component, or system is estimated to be able to function

SCE Southern California Edison

SCG Southern California Gas

SDG&E San Diego Gas and Electric

SEM Strategic energy management

QA/QC Quality assurance and quality control

UTC Use and technology categories


SBW Consulting, Inc. 1

Executive Summary

In this workplan, we—the SBW team—describe how we will complete the ex post evaluation of

custom-project claims that were reported by program administrators (PAs) for 2018 and 2019.

We have designed these evaluations to provide reliable estimates of life-cycle net kWh, kW and

therm savings for the custom-project portfolio. A separate workplan covers the portion these

savings that are attributable to the Industrial Strategic Energy Management (SEM) program.

For each PA we have defined specific sample domains based on the project’s classification using

three factors: sector (industrial, agricultural, and large commercial), custom-project type (new

construction or retrofit), and fuel type (electric or gas). Within each of these sample domains we

will estimate the life-cycle net savings and the life-cycle costs and benefits for each PA. We will

also provide actionable recommendations to improve future ex post evaluation methods and to

improve the PA’s savings claims for future custom projects.

Sample Design

Table 1 shows our sample design for estimating gross savings for the 2018 evaluation. Our

design represents the electric and gas savings claims for each PA with a sampling precision that

exceeds ±10% at 90% confidence in all cases. The sampled projects account for 51% of reported

kWh savings and 82% of reported therm savings. Additional projects will be selected within

each sample domain to meet our sample size objectives for net-to-gross ratio (NTGR) surveys.

We will use a similar design for 2019, and we will select projects at the end of each calendar

quarter. This will provide additional time for data collection and allow us to conduct the NTGR

surveys with decision-makers closer to the times that they made their decisions to proceed with

their projects. It will also enable us to adjust the sampling domains to adequately represent new

types of projects, such as commercial normalized metered energy consumption (NMEC), which

may become more important over the course of the evaluation.

Table 1: Sample Design for Gross Savings Evaluation of 2018 Custom Projects

Electric Sample Design Gas Sample Design

PA Project Count

Sample Count

Percent kWh

Savings Sampled

Percent kW

Savings Sampled

Relative Precision

Project Count

Sample Count

Percent Savings

Sampled

Relative Precision

MCE 40 14 70% 53% 10% NA NA NA NA

PG&E 2,004 56 47% 53% 7% 89 46 80% 7%

SCE 291 41 59% 56% 6% NA NA NA NA

SCG NA NA NA NA NA 91 26 87% 8%

SDG&E 114 29 63% 72% 9% 49 23 82% 10%

Total 2,449 140 51% 55% 5% 229 95 82% 5%



Methodology

We will conduct the 2018 and 2019 evaluations in two phases. In the first phase, we will review

all reported custom-project claims and make corrections as necessary to bring them into

compliance with California Public Utilities Commission (CPUC) policies. We will then

estimate the adjusted gross and net savings due to the corrections in this review. In the second

phase, we will conduct site inspections, interview decision-makers with NTGR surveys, and

review submetering and other types of data collection to support our estimates of ex post gross

and net savings for the sampled projects. Using the sample results, we will then estimate the ex

post life-cycle net savings and the cost-effectiveness for each sample domain for every PA.

Schedule

Figure 1 shows our schedule for completing this work.

Figure 1: Schedule for Ex Post Evalution of 2018 and 2019 Custom Projects

Year:

Deliverable Task / Subtask Due date Qtr: 1 2 3 4 1 2 3 4 1 2 3 4

Evaluation work plan . 2

Draft release - ED review 20-May-19 2

Draft release - stakeholder review 7-Jun-19 2

Final posted 6-Jul-19 2

Plan update Apr-20 2

Data collection and sampling plan Jul-19 . 2

2018 project data requests Jun-19 2

2019 project data requests Jul-19 2 2 2

Gross savings estimates

2018 reviews, data collection, analysis Dec-19 . .

2019 reviews, data collection, analysis Dec-20 . . . . .

Net savings estimates



Final evaluation report

2018

Population impact analysis Dec-19 .

Draft release - ED review 2-Jan-20 2

Draft release - stakeholder review 28-Feb-20 2

Final posted 30-Mar-20 2

2019





Data transfer and documentation

2018 Apr-20 2

2019 Apr-21 2

Key:

Activity to complete deliverable expected

Major deliverable to be completed

2019 2020 2021



1 Background

In this section, we summarize our understanding of the findings from prior evaluations of

custom projects, issues raised in recent energy savings performance incentive (ESPI) memos,

and relevant CPUC policies and guidance.

1.1 Prior Evaluations

Custom programs were evaluated in 2013, 2014, and 2015.1 The results of these evaluations

were remarkably similar. Gross energy-realization rates and NTGRs consistently hovered

between 0.5 and 0.6. The reasons cited for the relatively low realization rates and NTGRs

remained consistent across the studies. The primary reasons include:

� Errors in the calculation methods of the PAs

� Inappropriate baseline specifications

� Savings claims for ineligible projects

� Use of default operating conditions instead of forecast conditions or observed operating

conditions

Recommendations across the studies paralleled these reasons and included:

� The IOUs should improve calculation methods and protocols to increase the accuracy of

savings estimates.

� The IOUs should improve the documentation and reporting of the effective useful life (EUL)

of projects and the remaining useful life (RUL) of early-retirement (ER) projects.

� Improve the quality control of project operating conditions, ex ante baseline determinations,

savings calculations, and eligibility rules.

� Adjust project savings based on post-installation inspections and measurement and

verification (M&V).

� PAs should consider adopting program-implementation procedures and features designed to

increase program influence.

1.2 Issues Identified in ESPI Performance Memos

Issues identified during ex-ante review (EAR) were documented in the ESPI memos. Recurring

themes across PAs and program years included:

1 The 2013 study separated custom retrofit and nonresidential new construction (NRNC) projects as separate reports, while the 2014 and 2015 reports combined custom retrofit and NRNC projects.



� Calculation tools. Issues with the EnergyPro2 software used to calculate savings for the

Savings by Design program were identified on projects submitted by PG&E, SCE and

SDG&E. Similar issues occurred across multiple program cycles.

� Quality assurance/quality control (QA/QC). Persistent issues with internal due diligence

review were identified across all PAs.

� Measure life. Persistent problems with the correct assignment of measure effective useful

life and remaining useful life for accelerated replacement projects were identified. EULs for

retrofit add-on measures that exceed the RUL of the host equipment were also identified.

� Program influence documentation. Inadequate documentation of program influence was a

persistent problem across all PAs.

� M&V. EAR dispositions required additional M&V on multiple projects and measure types

for which measured performance data was needed to estimate savings. Measures requiring

additional M&V included: guest room energy management systems, non-DEER lighting

projects, and first-time projects with little performance history.

� Problematic measures. Issues were identified with savings calculations for several measure

types, requiring improvements to calculations, input parameters, or documentation. These

measures included: chiller optimization, air compressor plant projects, oil pumping,

monitoring-based commissioning, variable air volume conversions, and waste water

treatment plants.

1.3 CPUC Policies and Guidance

An important goal of this evaluation will be to determine whether the utility-reported energy

savings for custom projects in 2018 and 2019 conform to applicable CPUC policies and

guidance. As described in section 3, we will review all reported claims to determine whether

they conform to the policies and guidance provided the following documents:

� CPUC Energy Efficiency Policy and Procedures Manual (v. 5 or the most recent version) 3

� Utility Statewide Custom Policy and Procedures manual4

� Statewide Savings by Design Policy and Procedures manual5

� PA-specific policy and procedures manuals

� Industry standard-practice (ISP) studies completed and adopted before or during 2017

2 EnergyPro is building energy performance modeling software, sold by EnergySoft LLC, which is approved by the

California Energy Commission for demonstrating performance compliance with the nonresidential provisions of the California Building Energy Efficiency Standards.

3 CPUC. 2013. Energy Efficiency Policy Manual Version 5.0 For Post-2012 Programs. San Francisco, CA: CPUC. 4 PG&E, SCE, SoCalGas, SDG&E. 2018. 2018 Statewide Customized Offering Procedures Manual for Business. San

Francisco, CA: CPUC. 5 CPUC. 2018. Savings by Design 2018 Participant Handbook. San Francisco, CA: CPUC.



� Title 20 and 24 requirements in place when projects were permitted.

� CPUC policy papers and state-government memos addressing topics such as non-IOU fuel

sources and the requirements for preponderance of evidence for EUL/RUL.

� CPUC resolution E-4818 regarding assignment of project baselines

� Review dispositions of custom projects issued by CPUC staff

� NTG ratio and EUL/RUL support tables downloaded from READI v.2.5.1 available at

http://www.deeresources.com/index.php/deer-versions/readi



2 Evaluation Objectives

This workplan will meet the following objectives for the 2018 and 2019 program years:

� Gross Savings. We will estimate life-cycle gross kWh, kW and therm savings for the

custom-project portfolio, excluding the portion attributable to the Industrial SEM program,

which is covered under a separate workplan. As described in section 4, for each PA we will

define sampling domains based on three factors: sector (industrial, agricultural, and large

commercial), custom-project type (new construction or retrofit), and fuel type (electric or

gas). Within each of these sampling domains we will estimate gross savings for each PA.

During 2019, other domains may be added if new types of projects, such as commercial

NMEC, become an important part of the custom-project portfolio.

� Net Savings. We will estimate a net-to-gross ratio that can be multiplied by gross savings to

calculate the net savings attributable to custom projects within each domain. Market effects

will be accounted for by adding a market effects adder of 0.05 to all NTGR values, as

directed by the CPUC.

� Cost-Effectiveness. We will estimate life-cycle costs and benefits for each domain, for each

PA, and for the overall custom-project portfolio, based on incremental costs and the most

appropriate effective useful life and load shape.6 For early-retirement efficiency measures,

we will account for remaining useful life when we estimate life-cycle costs and benefits.

� Reproducible Results. We will document all our primary data collection efforts, modelling,

and data processing procedures to ensure that our results are transparent and can be

reproduced by other parties.

� Recommendations. We will develop actionable recommendations designed to improve the

PA savings claims for future custom projects and to improve future ex post evaluation

methods.

6 Load shapes will be selected from those shapes supported by the CPUC cost-effectiveness tool.



3 Review of Claims Database

This section describes the first phase of our evaluation in which we will review all savings claims

associated with custom measures reported by the PAs for 2018 and 2019 (The second phase of

our evaluation will focus on a sample of projects as described in sections 4 and 5.) In this phase,

we will examine all savings claims to determine whether these claims conform to CPUC policies

and guidance. The following subsections describe the criteria we will use to determine if savings

claims comply with CPUC polices and guidance, and the procedures for correcting claims that

do not comply.

3.1 Data Requests

Our review will be limited to claims data that is submitted by the PAs to CEDARS and other

information that it is practical for the PAs to provide for all their claims. Other information

may include clarification of installation dates, indication of hard-to-reach customers, values for

missing NTG or EUL IDs, second baseline savings, and clarification of measure descriptions.

3.1.1 Review Project Claims

We will review these claims to identify whether additional information is required to support a

review of compliance with CPUC policies. We will develop requests for additional information

as identified in this review.

3.1.2 Request Supplemental Data

Our data requests will address any critical missing information that could reasonably be

provided. Based on a recent review of claims for 2017, we anticipate requesting the following

information from the PAs.

� Installation Date. For claims with installation dates prior to the 2018 or 2019 program

years, we will request verification that the claims were delayed due the time needed to

complete M&V after installation. For claims with dates occurring after the program year, we

will request explanations regarding why the claims were not held and claimed for that

program year.

� Hard to Reach. For claims associated with hard-to-reach customers, we will request

identifying documentation.

� DEER NTG and EUL IDs. For claims missing either the DEER NTG or EUL ID, we will

request those ID values.

� Second Baseline Savings. For claims whose measure application type indicates an early-

retirement measure, we will request any missing savings for the second baseline.



� Measure Description. For claims with ambiguous or vague descriptions, we will request

clarification.

3.2 Review Reported Claims

3.2.1 Correct Claims Based on Supplemental Data

We will create an updated version of the reported claims based on the responses to the

supplemental data requests by either correcting values provided to CEDARS or adding elements

of data not included in CEDARS.

3.2.2 Claim Year

The CPUC directed the PAs to include only savings for measures that were installed in the same

program year for which they are reporting. In 2017, the CPUC provided further guidance to the

PAs that savings applied towards ESPI payments will only be counted for measures that were

installed in the program year for which ESPI payments are being requested. The guidance

requires the PAs to identify the installation year for all projects that are included in their claims

for that program year. Additionally, the CPUC directed PAs to identify claims with justifiable

exceptions to the rule regarding installation-dates. Unless sufficient documentation supports an

appropriate exception, any claimed savings occurring outside of the program year will be set to

zero, effectively removing them from the claim.

3.2.3 Classification Based on Measure Description

We will examine information in the measure description field of the claims database and correct

errors in the values of the following fields: use category, use subcategory, tech group, and tech

type.7 These corrections will result in updates to the values for NTGR, EUL, and RUL.

Measure descriptions will be used to establish each claim’s NTG, EUL, RUL, and measure

application type. We will match the measure descriptions for each claim with the DEER ex ante

NTG, EUL and RUL values8 that were adopted for use in 2018 or 2019. This will enable us to

map measure descriptions to values for specific use category, use subcategory, tech group, and

tech type. These are referred to collectively as UTCs, which is an abbreviation for use and

technology categories9.

7 The CPUC’s Remote Ex Ante Database Interface (READI) utilizes use category and use subcategory to define how or

where a technology is used. All technologies that are referenced in measure definitions are further organized into tech groups and tech-types (i.e., a subcategory of tech group).

8 NTGR and Full EUL and RUL support tables downloaded from READI v.2.5.1 available at: http://www.deeresources.com/index.php/deer-versions/readi

9 Tables defining each of the UTCs (use category, use subcategory, tech group and tech type) are available from READI v.2.5.1 available at: http://www.deeresources.com/index.php/deer-versions/readi



To ensure a claim is correctly categorized according to its UTCs, we will develop a keyword list

of names and terms that are common to each measure description use category. This will

provide the highest level of classification for claims and will be used to further match a claim to

an accepted value in the ex ante database. We will develop the keyword list from the measure

description from each claim and the use categories contained in the ex ante database10 measure

list that was adopted for use in 2018 and 2019. Table 2 gives examples of keyword matches.

Table 2: Example of Keyword List

Ex Ante Database Use Category Claims Database Measure Description Keywords

HVAC (HVAC) Chiller, HP, heat pump, package AC

Appliance or plug load (APPPLUG)

Television

Building envelope (BLDGENV)

Envelope, SHGC, insulation, weatherization, shell, windows, fenestration

To ensure the correct label is assigned, we will search the measure descriptions in the claims

database for common keywords and map them to the appropriate measure description use

category. If a claim is found to have a correct use category, that use category will be assigned

and used to further match a claim to remaining UTC values. If a claim has an incorrect use

category, a new one, based on the keyword mapping, will be assigned for use in further

matching the claim to the remaining UTC values. If a use category cannot be matched based on

information contained in the measure description, values for all four UTCs in the claims

database will be accepted as originally submitted in the claim.

Once a use category is matched for a claim, we will scan the measure description field again for

subsequent keywords associated with the use subcategory, tech group, and tech type, and then

correct those values in a similar manner until we successful match the claim to a unique

measure in the ex ante database.

Upon completion, all claims will be associated with four UTCs, either updated to a corrected

value based on our keyword mapping, or unmodified if we are unable to match a keyword in

the claim’s measure description to any of the four UTCs.

3.2.4 Net-to-Gross Ratios

The DEER NTG table provides approved NTGR values for kWh and therm savings that apply

to combinations of the following fields: building type, building vintage, measure delivery, sector,

use category, use subcategory, tech group, and tech type. We will update NTGR values for kWh

and therms based on corrected values for these fields. The corrected values will be used to find

10 READI - Remote Ex Ante Database Interface version 2.5.1. Available at ftp://deeresources.com/DEER/

READI_v2.5.1.zip



the appropriate NTGR value that applies to a unique measure in the ex ante database.We will

only apply NGTR values that the CPUC has adopted for use in the program year.

We will use the methodology described in section 3.2.3 to determine the correct UTCs for each

claim. Once we determine the correct UCTs, we will associate the updated NTGR values for

each unique measure using a lookup and match to a unique measure in the ex ante database.

To match unique measures in the DEER NTG values, we will rely on the corrected UTCs, the

listed delivery types or the NTG_IDs that are provided as part of the claim when they are found

to be correct. If a claim matches a single measure, the NTGRs for kWh and therms will be

assigned to that claim. Any claim not matched to a single row in the table will be left unchanged

with respect to the UTCs, NTG, and NTG_ID values.

3.2.5 Effective Useful Lifetimes

The DEER EUL table provides approved EUL values defined by various combinations of the

building type, building location, tech group, tech type, use category, use subcategory, and

sector. We will update EUL and RUL values as appropriate based on any corrected UTC fields

that define those values in the ex ante measure database. We will only apply EUL and RUL

values that were adopted for use in the program year.

In a manner like the methodology that we will employ for measure descriptions, we will

determine the correct UTCs for each claim and associate the correct EUL/RUL values

matching a unique measure in the ex ante measure database.

To match corrected EUL and RUL values with a unique measure in the ex ante measure

database, we will rely on the corrected UTCs, the listed delivery type, or the EUL_ID that is

provided with the claim when it is found to be correct. If a claim matches a unique measure in

the ex ante database, we will assign the EUL, RUL, EUL_ID, and associated UTCs to that

claim. Any claim not matched to a single row in the table will be left unchanged with respect to

the UTCs, EUL, and EUL_ID values.

Measures with measure application type set to early retirement are required to have a value

assigned for the RUL. We will update the RUL values to be consistent with corrected measure

application type and corrected EUL by matching claims to a unique measure in the ex ante

measure database.

3.3 Calculate Lifetime Net Savings

Based on the database review adjustments described above, we will calculate net life-cycle

savings for kWh, kW and therms for each claim.

For claims associated with early-retirement measures, we will calculate saving as follows:

Net Life-cycle Savings =

[(BS1 × RUL) + (BS2 × [EUL − RUL])] × (NTGR + MEA) × GRR × IR



where:

BS1 = first baseline savings

RUL = remaining useful life

BS2 = second baseline savings

EUL = effective useful life

NTGR = net-to-gross ratio

MEA = market effects adder

GRR = gross realization rate

IR = installation rate

For all other measures, savings will be calculated as follows:

Net Life-cycle Savings = (BS1 × EUL) × (NTGR + MEA) × GRR × IR

where:

BS1 = first baseline savings

EUL = effective useful life

NTGR = net-to-gross ratio

MEA = market effects adder

GRR = gross realization rate

IR = installation rate

The market effects adder for both formulas is 0.05.11

When we report savings from this evaluation, we will show the corrections associated with our

review of the claims database separately from the combined impact of the database review and

ex post evaluation of savings described in section 5.

11 CPUC. 2012. D.12-11-015 Decision Approving 2013-2014 Energy Efficiency Programs and Budgets. San Francisco, CA:

CPUC.



4 Sample Design

In this section we describe our design for sampling projects for ex post gross and net savings

evaluation.

4.1 Gross Savings Sample for 2018

4.1.1 Sample Frame

The custom measures table located on CEDARS provides the foundation for the 2018

Industrial, Agricultural, Large Commercial (IALC) sample frame. This table also includes

claims from codes & standards and residential programs, both of which we excluded from the

sample frame. The remaining claims are subject to this IALC ex post savings evaluation.

To form sample domains, we first assigned the claims to subject areas based on whether they

were from a Savings By Design (SBD) or other New Construction (NC) program, or a program

targeting retrofit of existing facilities. We labeled the former as SBD/NC and the latter as

Existing Retrofit. For the purpose of the IALC evaluation we call these subject areas Impact

Type. Then we categorized the claims by Program Administrator (PA) and by sector. Thus the

sample domains are by PA-Impact Type-Sector.

We also segmented the population by fuel – Electric or Gas. First, we collapsed the claims to

project level using unique project identifiers in the CEDARS data, i.e., ProjectID or

ApplicationCode. Then because most claims have savings in both fuels, even for the single fuel

utilities, we designated whether a project was primarily an electric savings project, a gas savings

project, or a combination. We used the following logic to make this determination12. If PA is:

� MCE or SCE, then we assigned the project to the electric sample frame

� SCG, then we assigned the project to the gas sample frame

� PG&E or SDG&E and project has

� positive electric savings and negative gas savings, then we assigned the project to the

electric sample frame

� positive gas savings and negative electric savings, then we assigned the project to the gas

sample frame

� positive electric and positive gas savings, then we assigned the project to both the electric

sample frame and gas sample frame

12 There were no projects with both negative electric and negative gas savings.



Regardless of the fuel we assigned the project to, we will treat all savings associated with all

projects in each frame. The claims aggregated to project-level form the records in the sample

frame for each fuel. Table 3 displays a summary of the 2018 IALC frame by sample domain.

Table 3: 2018 IALC Sample Frame Summary

Sample Domain

Electric Lifecycle Net Savings

Gas Lifecycle

Net Savings

Total Project Count

Electric Project Count

Gas Project Count PA Impact Type Sector

(GWh) (GW) (kBtu)

MCE Retrofit COM 9,997 1 -35 40 40 NA

PGE SBD/NC AG 17,945 4 6 2 2 1

COM 117,493 22 1,766 24 20 15

IND 2,903 0.5 34 2 2 2

Retrofit AG 127,178 13 1,575 137 134 4

COM 450,256 50 9,053 1,671 1,664 53

IND 393,052 45 9,601 190 182 14

SCE SBD/NC COM 58,009 9 577 18 18 NA

Retrofit AG 16,467 1 0 20 20 NA

COM 205,372 22 386 207 207 NA

IND 72,097 6 -4 46 46 NA

SCG Retrofit AG 0 0 3,049 1 NA 1

COM 0 0 3,439 67 NA 67

IND 0 0 6,334 23 NA 23

SDGE SBD/NC COM 68,124 11 1,056 42 42 29

Retrofit COM 81,411 7 3,010 69 66 19

IND 9,758 1 90 6 6 1 1,630,061 192 39,939 2,565 2,449 229

4.1.2 Sample Domains, Size and Expected Precision

Based on the distribution of projects and savings in the 2018 program year shown in Table 3

above, we developed a sample design shown in Table 4 and Table 5 below. For the estimated

gross savings within each domain, we have targeted the 90% level of confidence with a relative

precision of plus or minus 15% or better while for the estimated gross savings within each PA,

we have targeted the 90% level of confidence with a relative precision of plus or minus 10% or

better.



Within each domain, we selected a stratified random sample13 that minimizes the required

sample size to achieve the targeted levels of confidence and precision. Such a design can provide

the same level of confidence and precision as a simple random sample but with a smaller sample

size. Furthermore, each domain has a certainty stratum containing the largest savings project(s)

in that domain. The projects in the certainty strata have a probability of selection of 1 and,

therefore, no sampling error. Also, most domains have an exclusion stratum containing the

smallest savers which are excluded from the sampled strata. Our use of exclusion and certainty

strata further reduced sample sizes, while maintaining the confidence and precision targets.

Table 4 and Table 5 present the target sample sizes for each domain in the electric and gas

frames, respectively. For domains with a small number of projects, i.e., fewer than 10, we

selected all projects. For example, for the domain defined as PGE-SBD/NC-AG, which has

only two projects, both projects were selected. For these domains, as in the case of the certainty

strata, there is, by definition, no sampling error. Fifteen projects are in common between the

electric and gas samples. Each of these 15 projects count toward the required sample size in

their respective fuel frames.

Table 4: 2018 IALC Sample Design – Electric Frame

Sample Domain Total Project Count

Sample Size

Total LifeCycle Net Savings

(GWh)

Sampled Lifecycle Net

Savings (GWh)

Percent of Savings in

Sample

Target Relative Precision PA

Impact Type

Sector

MCE Retrofit COM 40 14 9,997 7,018 70% 10%

PGE SBD/NC AG 2 2 17,945 17,945 100% 0%

COM 20 5 117,955 86,138 73% 9%

IND 2 2 2,903 2,903 100% 0%

Retrofit AG 134 12 127,178 69,312 54% 14%

COM 1,664 24 450,481 83,685 19% 10%

IND 182 11 394,325 264,690 67% 15%

SCE SBD/NC COM 18 8 58,009 41,082 71% 13%

Retrofit AG 20 10 16,467 11,941 73% 8%

COM 207 14 205,372 129,651 63% 9%

IND 46 9 72,097 23,603 33% 12%

SDGE SBD/NC COM 42 15 68,124 41,695 61% 13%

Retrofit COM 66 8 81,411 48,988 60% 14%

IND 6 6 9,758 9,758 100% 0%

Total 2,449 140 1,632,023 838,409 51%

13 Stratified random sampling is a method of sampling that involves the division of a population or subpopulation

(domain) into smaller sub-groups known as strata. In stratified random sampling, the strata are formed based on

members' shared attributes or characteristics such as income or educational attainment, or in our case, savings of similar size. See Chapter 5 of Sampling Techniques, Cochran, (1977).



Table 5: 2018 IALC Sample Design – Gas Frame

Sample Domain Total Project Count

Sample Size

Total Lifecycle Net Savings

(kBtu)

Sampled Lifecycle Net

Savings (kBtu)

Percent of Savings in

Sample

Target Relative Precision PA

Impact Type

Sector

PGE SBD/NC AG 1 1 6 6 100% 0%

COM 15 8 1,773 1,509 85% 14%

IND 2 2 34 34 100% 0%

Retrofit AG 4 4 1,584 1,584 100% 0%

COM 53 23 9,855 8,434 86% 10%

IND 14 8 9,627 6,720 70% 13%

SCG Retrofit AG 1 1 3,049 3,049 100% 0%

COM 67 13 3,439 2,271 66% 13%

IND 23 12 6,334 5,835 92% 15%

SDGE SBD/NC COM 29 16 1,056 979 93% 15%

Retrofit COM 19 6 3,010 2,334 78% 12%

IND 1 1 90 90 100% 0%

Total 229 95 39,857 32,844 82%

As shown in Table 4 and Table 5, this sample design of 140 electric projects and 95 gas projects,

for a total sample size of 235, touches 51% of claimed electric savings and 82% of claimed gas

savings. Table 6 summarizes the sample design by PA for the electric frame followed by Table 7

for the gas frame.

Table 6: 2018 IALC Sample Design Summarized by Program Administrator – Electric Frame

PA Project Count

Sample Count

Total Lifecycle Net Savings

Sample Lifecycle Net Savings Percent Savings

Sampled Sample Relative

Precision GWh GW GWh GW

MCE 40 14 9,997 1 7,018 1 70% 10%

PGE 2,004 56 1,110,788 136 524,672 72 47% 7%

SCE 291 41 351,944 38 206,277 21 59% 6%

SDGE 114 29 159,293 18 100,442 13 63% 9%

Total 2,449 140 1,632,023 193 838,409 106 51% 5%

Table 7: 2018 IALC Sample Design Summarized by Program Administrator – Gas Frame

PA Project Count

Sample Count

Lifecycle Net Savings (kBtu)

Sample Lifecycle Net Savings (kBtu)

Percent Savings Sampled

Sample Relative Precision

PGE 89 46 22,878,405 18,286,192 80% 7%

SCG 91 26 12,822,855 11,155,400 87% 8%

SDGE 49 23 4,155,980 3,402,752 82% 10%

Total 229 95 39,857,240 32,844,345 82% 5%



We will oversample in each of the sampled strata in all domains to ensure that we have

replacements in case customer representatives from our primary sample selections are not

responsive. We will carefully monitor sample disposition throughout the evaluation to ensure

we are closely following the sample design to produce reliable results.

4.2 Net Savings Sample for 2018

We will conduct NTGR survey interviews with customer decision-makers from the projects in

the sample described above. We will randomly select additional projects within each domain

and strata to complete a total of 285 NTGR survey interviews.

4.3 Gross and Net Savings Sample for 2019

We will select samples after the end of each calendar quarter to represent the custom projects

installed in 2019. By sampling each quarter, we will

1. Allow more time for data collection, which will be especially beneficial for those projects

requiring sub-metering of affected systems or equipment.

2. Conduct the NTGR survey with decision-makers closer to the time that the decision was

made to proceed with each project.

3. Adjust the sampling domains to adequately represent new types of projects, such as

commercial NMEC, which may become more important over the course of the year.

Our gross and net samples sizes for the year in total, will again be 235 and 285 respectively, and

we expect to use a sample design like the one we described for 2018. We will develop a

quarterly forecast of completed projects by domain, based on the annual pattern of project

installation during 2017 and 2018, to guide the allocation of the sample across the quarters. We

expect somewhat lower sampling precision from this quarterly design, but we believe that

disadvantage is outweighed by the benefits listed above.



5 Ex Post Evaluation Methods

This section summarizes our ex post gross and net evaluation methods. This section also

describes our approach to analyzing the measure costs, lifetimes, and load shapes needed to

assess cost-effectiveness in section 6.

5.1 Data Requests

We will issue a series of data requests to the PAs for the files and other information needed to

complete ex post evaluation of the gross and net samples for 2018 and 2019.

5.1.1 Obtain Project Files

We will request that the PAs upload all files for the sampled projects to the evaluation,

measurement and verification (EM&V) platform on DEERresources.info. We will retrieve the

uploaded files and move them to secure storage that is accessible to our analysts. We will review

the project files to determine whether they are missing any critical data, models, or

documentation.

5.1.2 Request Missing Data, Models, or Documents

We will use the EM&V platform to request any missing critical data, models, or documentation

for sampled projects. Based on our recent experience reviewing a sample of 2017 projects and

on our experience with previous ex post evaluations, we anticipate that these requests may

include:

� Working calculators. Spreadsheet models or hourly model input files.

� Fuel switching justification. If a retrofit project results in an energy consumption shift from

electricity to gas or vice versa, then the three-prong test14 is required to assess eligibility for

switching fuel.

� On-site generation analysis. Any on-site generation, such as cogeneration or behind the

meter solar arrays at the site, requires analysis that considers the marginal change in IOU-

supplied fuels resulting from the project.

� Justification of early retirement and remaining useful life. Early-retirement projects

require evidence that the existing equipment is functional and could continue to be used for

the claimed remaining useful life.

� Evidence of program influence. Eligible projects require evidence that project initiation was

influenced by the program.

14 Energy Efficiency Policy Manual Version 5, page 24, Cost Effectiveness Rule XV.10



� Permitting. If permits are required for measure installation, the customer/installer must

obtain such permits.

� Contacts. Names and contact information for decision-makers or other parties familiar with

the systems, equipment or practices affected by the project.

� M&V data. Trend logs, interval data, billing energy use data, or other metering records that

were obtained by the program during the baseline or post-installation periods. This would

include any test results associated with commissioning the affected systems or equipment.

� Affected systems, equipment or practices. Clarification of the characteristics of the systems,

equipment, or practices that were affected by the project, including both baseline and post-

installation conditions.

� Costs. Invoices or other related cost information used to determine the measure cost.

5.2 Gross Savings Evaluation

This section describes how we will develop site-specific measurement and verification plans for

each sampled project, including special circumstances for new construction and site-specific

NMEC projects. This section also discusses how our team will determine ex post conditions and

use those findings to re-estimate project gross savings.

5.2.1 Prepare Site-Specific M&V Plans

We will develop site-specific M&V plans to guide the NTGR interviews, site visits, data

collection efforts, and the procedures for estimating savings. These M&V plans will also serve as

the basis for the final project-specific M&V reports. To speed the development of these

individualized M&V plans, we’ll begin by preparing a template for a general M&V plan for

common project types that we can then customize to accommodate more unique project types.

We will submit draft and final templates to CPUC staff for review and approval.

These general templates will be well-suited to projects that feature custom capital retrofit and

operational measures in the industrial, agricultural and large commercial sectors. M&V plans

for such projects must deal with questions regarding normal and early replacement, and they

must apply International Performance Measurement and Verification Protocol (IPMVP)

Options A, B or D in their approach. However, custom projects in the following two subject

areas also have unique attributes that must be dealt with somewhat differently:

� New construction capital measures (Savings by Design)

� Site-specific NMEC, which includes high opportunity projects and programs (HOPPs).

We discuss the special approaches for these subject areas in section 5.2.1.2 and section 5.2.1.3,

respectively. To avoid scattering these topics across many subsections, each section instead

treats these topics comprehensively by fully describing the specific issues and unique situations

that affect M&V planning, data collection, and analysis.



5.2.1.1 Features of All M&V Plans

Once we have selected a sample we will use the approved template to create customized M&V

plans for each sampled project. These customized M&V plans will serve as roadmaps guiding

the NTGR interviews, site visits, data collection, and the procedures for estimating savings. Our

team will also standardize the processes for plan development, technical support, and quality

assurance to ensure that each site-specific M&V plan conforms to this workplan, meets quality

control standards, and remains consistent with other similar plans that have already been

approved.

After we review the project documentation and fully understand the nature of the implemented

measure(s), one of the first steps in M&V planning will be to recruit the customer for the M&V

effort. Doing so dramatically minimizes the chance that customers will not cooperate and

therefore minimizes the likelihood of any time wasted on planning efforts. Early customer

contact also enables the person in charge of leading the M&V plan to collect key logistical

information to inform the approach to the plan by better understanding the type of data that will

be available for the project and whether onsite data logging is appropriate. We will use

information obtained from the customer contacts to refine the M&V plans prior to the site visits.

During our conversations with customers we will also verify that the equipment has been

commissioned and that it is operating as defined in the project documentation. If any significant

changes have occurred since the time that the measure was implemented, we will make

appropriate adjustments to our M&V planning.

A critical aspect of the M&V planning will be to ensure that the approaches we follow are

consistent with the guidelines developed for the ex ante reviews of the projects. The rulebook

developed for the ex ante reviews contains key guidance, rulings, and decisions, and it will be an

important resource for developing M&V plans.

In a similar vein, we will coordinate with the team working on the Group A deemed evaluation

to ensure that both groups are applying consistent approaches for evaluating HVAC and lighting

measures. We will also work with the Group A team to identify any customers that both groups

have sampled so we can coordinate and minimize customer contacts. The best time to

coordinate these efforts will be immediately after the sample is drawn.

We will develop a comprehensive training regimen to ensure that all field staff are

knowledgeable about M&V planning, field procedures, and the proper use of the data collection

forms. This will ensure effective data collection and compliance with the evaluation protocols

and CPUC guidance. We will also develop a handbook and provide real-time support while our

staff are in the field to ensure their questions and issues can be addressed promptly.

Each plan will address the key questions listed below.

Is the project eligible?

Before developing a full-fledged customized M&V plan for a project, we will request project

documentation as described in section 5.1.2 and perform an initial project review to assess



project eligibility. While this review will not include an assessment of free-ridership, which will

be covered elsewhere, it will look for relevant information, such as projects that were completed

outside of the program year without corresponding M&V and fuel switching that fails the three-

prong test. If we determine that a project is ineligible, we avoid spending time developing

unnecessary plan elements.

How will we analyze savings?

As a starting place for developing our customized M&V plans, we will review the methodology

used for analyzing the ex ante savings estimates. In many cases, we will pursue a similar

calculation strategy using updated data to determine ex post savings. However, depending upon

the available data, we may determine that an alternative evaluation methodology will yield

more accurate savings results. In some cases, alternative evaluation methodologies may also be

dictated by data availability at the time of the evaluation.

We will identify any parameters with substantial uncertainty and determine whether contact

with the customer or vendor is needed to mitigate this uncertainty. We will also determine

whether it is sufficient to rely on a phone call or email, or whether it is necessary to conduct an

onsite visit for inspections, measurements, or metering.

Implicit in our review and corresponding planning for data collection is establishing whether the

project falls into the basic or enhanced rigor level.15 These indicate the level of resources and

effort required to achieve reliable evaluation savings estimates for that project. Necessary

resources depend on project characteristics, which can be difficult to predict in advance. For

example, a large, complex project may be easy to evaluate because it has well-documented

models and abundant data from before and after the measure installation. Conversely, a small,

simple project may require more resources to evaluate because it lacks sufficient data and it

requires a new analysis approach that must be developed from scratch.

What data should we collect?

Project M&V leads will develop data collection forms and survey instruments that are

customized according to the site-specific M&V plan. For example, commercial building projects

may require onsite survey forms to verify and/or update the building characteristics and

operations data that are used to drive building simulations, while industrial process projects may

require customized data collection forms to suit the unique needs of the evaluation. In instances

where a project contains early-retirement-measure application types, there may be claims for

dual baselines, and the plan will necessarily address estimating savings for both.

Typical data collected for projects will include:

� Verification of installation and operation of the specified equipment

15 Rigor level is defined for program evaluations in the 2006 California Energy Efficiency Evaluation Protocols. For

engineering models, these protocols call out IPMVP Option A (Partial Retrofit Isolation) for basic rigor, and Option B

(Retrofit Isolation) for enhanced rigor. Our approach is consistent with this general philosophy, applied to specific projects and measures.



� Operational conditions, including load, hours of use, and seasonal variations

� Power measurements of equipment that are taken over a representative range of operating

conditions and a period of time that is long enough to establish normal operational

parameters with a high degree of certainty

� Trend data from onsite monitoring systems or building management systems that show

equipment operation

� Production data, if equipment operation is directly related to production

Often the ex ante project files contain data collected from the site, including facility trend data,

baseline conditions, and operation immediately after project installation. In addition to the ex

ante data, we have found it is best to collect updated trend data from the site since conditions

sometimes change, either in response to operational changes or seasonal variations. The best

estimates of long-term energy use can often be obtained by combining data from the ex ante

analysis with data collected during an evaluation site visit.

What about net savings?

Projects will also be sampled separately in the net savings evaluation described in section 0.

That section describes our plans for customizing or adapting survey scripts and interview guides

to estimate free-ridership for these projects.

5.2.1.2 New Construction

This evaluation will account for any unique aspects of new construction projects within the site-

specific M&V planning described above. New construction projects under Savings by Design

(SBD) cover a wide variety of sectors and industries, including green-field new construction, as

well as major expansions and renovations to existing buildings and industrial processes. SBD

offers both a whole building approach and a systems approach to commercial new construction

projects.

Incentives are paid based on energy savings calculated using an approved calculation tool.16 The

whole building approach uses compliance software to estimate the compliance margin (savings

above code) and calculate incentives. Projects permitted under Title 24, versions 2013 and

newer, use compliance tools based on software from CBECC-Com: California Building Energy

Code Compliance (for commercial/nonresidential buildings), which uses EnergyPlus as the

simulation engine. Previous rounds of the IALC evaluation sampled buildings that were

permitted under the 2008 standards that used an EnergyPro simulation engine driven by

DOE2.1E software. Buildings to be sampled for this evaluation likely received permits under

Title 24, versions 2013 and newer. Our evaluation data collection and software interface

processes will adapt to this new simulation environment.

16 One example of a SBD tool is EnergyPro Version 7, a software package approved by the California Energy

Commission for use with the whole building approach. For the systems approach, programs typically use simplified calculation approaches and tools, such as SimCalc.



In addition to Title 20 and Title 24, our M&V plans will consider regulations governing the

minimum code-compliant baselines for new construction projects. These include emission limits

imposed on industrial process equipment by the various air quality management districts

(AQMD) in the state. We will develop data requirements for documenting compliance with

AQMD regulations. Because high-efficiency industrial equipment may be needed to comply

with AQMD regulations, this may limit the energy savings options that are available beyond

code.

As part of our sampled project review, we will assess any calculation tools applicable to the

savings estimates for industrial process measures. The CPUC currently maintains the

Calculation Tool Archive. Our team will assess new processes and tools encountered during the

evaluation and add them to the archive as appropriate.

Some of the risks, challenges, and barriers associated with evaluating SBD projects include:

� Dealing with a particularly wide variety of commercial and industrial projects. For example,

industrial projects often feature unusual baselines and calculation methods that require

unconventional approaches to data collection.

� Using updated simulation software to estimate above code savings. New construction and

major renovations of commercial buildings are covered under Title 24, using compliance

software approved by the CPUC. Projects permitted under Title 24, versions 2013 and

newer, use EnergyPlus as the simulation engine, which requires different skills and tools.

� Obtaining sufficient project data to validate calculation tool inputs. This includes building

and project plans, equipment submittals in electronic format, and supporting spreadsheets

with “live” calculation cells.

5.2.1.3 Site-Specific NMEC Projects

Our evaluation planning will also address the unique aspects of NMEC projects within the site-

specific M&V planning explained above. To meet the evaluation goal of minimizing ex post

evaluation risk, we will confirm the availability of data at any point in the entire project life-

cycle. If possible, it will also be advantageous to have data to support parallel analyses to

triangulate estimates, using both top-down and bottom-up approaches.

The key to having consistent and sufficient data is to set up data acquisition requirements,

relationships, and processes as early as possible. The evaluation analyst will gather key data for

the relevant time periods (baseline or annual reporting period), such as:

1. Electric and/or gas usage (15-minute, hourly, or daily intervals) with corresponding

measurement boundaries,

2. Weather conditions,

3. Occupancy,

4. Ex ante savings,



5. Project-related activities and dates of activities, and

6. Non-routine events, that is, significant non-project related changes, such as tenant

improvements or other capital projects, setpoint or scheduling changes, occupancy changes,

or utility infrastructure failures.

Occupancy data can be particularly important for certain building types, such as large office

buildings, hotels, and hospitals. If occupancy data is planned for at the start, then it can be

included as an independent variable in the model. If not planned for, there is an increased

likelihood that any changes in occupancy would need to be treated as a non-routine event. For

building types that have frequent or ongoing changes in occupancy, such as hotels, it is

unacceptable to treat occupancy changes as non-routine.

For each sampled NMEC project, we will review the model from the project and create an

NMEC model using the ECAM™ tool. To prepare for analysis, we will clean all meter data

using statistical methods, and, if any data issues arise, we will confirm with onsite visual

verification. We will cross-check historical weather data using one or two additional weather

stations. We will develop a model for the reporting period based on meter and weather data. We

will calculate model fitness statistics on the reporting period model and compare them to the

criteria specified in the draft Guidance for Program Level M&V Plans: Normalized Metered

Energy Consumption Savings Estimation in Commercial Buildings17. If the reporting period

model does not meet desired model fitness, we will mention that information in our report. If

the reporting period model is acceptable, we will then apply typical weather to establish typical

reporting period consumption. If we identify any issues with any of the models in our analysis,

we will reconcile their results.

We will then estimate the normalized savings for the project by taking the difference between

baseline typical consumption and reporting period typical consumption. If significant non-

routine events occurred during the baseline or reporting phases, we will estimate the impact of

those events and adjust the savings for the project. If we find significant differences after

comparing our savings estimate to the program savings estimate for the project, then we will

investigate the reasons using a thorough process developed for the early M&V study of the

PG&E Commercial Whole Building Demonstration.18 We will report the magnitude of the

difference in savings estimates and provide explanations for those differences.

5.2.2 Determine Ex Post Conditions

As outlined in each site-specific M&V plan, we will determine ex post conditions, and, in some

instances, reconfirm baselines by conducting site visits and interviews with facility personnel

who are familiar with the sampled projects and site operations. Key elements of these efforts are

described below.

17 Lawrence Berkeley National Laboratory. 2018. Guidance for Program Level M&V Plans: Normalized Metered Energy

Consumption Savings Estimation in Commercial Buildings Version 1.0. San Francisco, CA: CPUC. 18 SBW. 2019. Commercial Whole Building Demonstration Early M&V Report. San Francisco, CA: PG&E..



Coordinating with the customer

As described above, we will contact the selected project sites as part of the M&V planning

process. After first attempting to reduce disruption to facility personnel by determining if any

projects share contacts or sites, our field engineers will coordinate with program implementers

to contact customers. Depending on the requirements in the site M&V plans, we will cluster

interviews and site visits by location to schedule them at times that create the least burden on

customer staff and at times during which data can be appropriately gathered, even if this means

that the interview or visit occurs outside normal working hours, or if we must wait until the

facility has downtime.

Similarly, to the extent possible we will maximize customer convenience and data security by

enabling them to fulfill data requests through secure electronic file exchanges. If we need to

obtain any other required information, we will continue our discussions with the customer

representative or members of the customer’s engineering or operations staff. Likewise, if we

need information from one of the vendors associated with the project, we will ask the customer

for the best way to contact that vendor, and then we will follow up with that contact.

At the outset of site visits, we will determine all safety and access requirements and make

necessary arrangements to conform to the customer’s requirements. In all cases, we will comply

with general OSHA safety regulations and NFPA70e requirements for arc flash protection when

supervising electrical power measurements.

Visiting project sites

We will visit customer sites to inspect the systems and equipment affected by their projects. The

primary objective of the inspection is to determine whether their project was completed and if it

is producing savings. During our site visit we will confirm the make and model of the project

equipment and determine whether its installation and operation conform to the project’s intent.

We will confirm current and anticipated equipment operation by interviewing the customer’s

staff and by inspecting equipment control settings. We will record our conversations and take

pictures to document the visit and facilitate quality control review by other members of our

team.

Based on the findings of our site visit and on our interviews, phone calls, and email exchanges

with customers and vendor staff, we will identify any appropriate corrections to savings and

cost-effectiveness parameters, and then we will document those differences in a manner similar

to that used during the file review.

While at the customer facility, we will collect data in accordance with the customized M&V

plan, adjusting our efforts as necessary based upon the operations found onsite and on actual

data. For example, we have found that sometimes our onsite customer contacts are uncertain of

the data trends in their systems, so it may be necessary to adjust our M&V plans based on actual

conditions found at the site. Our team is experienced at extracting data from a variety of

trending systems, and we frequently work with onsite personnel and third-party suppliers to



obtain trend data. However, if trend data is not available, we may have appropriate data logging

equipment installed by electricians after we receive approval from the customer.

Our team is well versed in the practice of data logging. We are familiar with the use of current-

only loggers, true-RMS power data loggers, light sensors, temperature and humidity loggers,

and various spot measurement equipment. Our team will rely on our best professional

judgement to determine appropriate conditions for each type of measurement and which

locations are best to deploy the equipment.

5.2.3 Re-Estimate Project Gross Savings

Once we have obtained the appropriate equipment readings or trend data for a site, we will

proceed to analyze the savings. For each sampled project we will re-estimate gross savings by

making one or more corrections to the inputs or algorithms of the ex ante model(s) that were

used by the PAs to estimate the original savings claim. These corrections will be based on the

findings from our determination of ex post conditions. The corrections will fall into the

following four categories.

1. Correct Algorithm Errors. These corrections could apply to custom spreadsheet models

where we find formula errors. It is beyond the scope of this effort to correct algorithmic

errors in building simulation programs or other specialized calculators such as AIRMaster or

MotorMaster.

2. Revise Algorithms. These revisions could apply to any type of model used to estimate

project savings. Our intent with these revisions is to achieve a model that more reliably

estimates the impact of the project on affected systems and equipment.

3. Modify Baseline Characteristics. These modifications could apply to any type of project,

ranging from minor changes to the capacity, efficiency, operating hours, or control

sequences of the affected equipment to major changes due to the imposition of code or

standard practice features. In cases where a project was misclassified as replace on burnout

and we need to model it as early retirement, this would also require us to establish the

characteristics of the second baseline, which would determine savings after the remaining

use life of the initial baseline conditions, i.e., at the future point in time where normal

replacement would occur. In addition, we would need to establish the justification for early

retirement by collecting customer statements that the existing equipment would have been

operational for the RUL period.

4. Modify Measure Characteristics. As with the baseline characteristics, these modifications

could apply to any type of project. They would reflect the capacity, efficiency, operating

hours, or control sequences verified via direct observation or through our contacts with

customer staff or vendors.

Each of these types of corrections and modifications will be applied in the sequence listed

above. Where appropriate, we will calibrate modelled energy use to post-retrofit billing records.

In cases where short-term metering occurred, that metering data will be incorporated into the



analysis. For sites where measure savings are relatively large compared to billed energy use, we

may estimate savings using billing regression techniques or we may compare pre and post

installation billing records with the savings estimates that were calculated by algorithm. We will

review savings at both the measure and project levels, and we will document any differences

between our evaluation savings estimates and original program savings estimates.

Typically, this analysis will be performed in Excel, although additional tools such as eQUEST

or R may be used if the complexity or volume of data requires more specialized analysis. For

projects that exhibit weather dependence, we will obtain local weather data and perform

regression analyses for baseline and efficient conditions, each of which will be normalized to

TMY3 data. For some industrial projects, such as compressed air, the use of operational

equipment may vary with production levels. In these cases, we will request long term

production or operations data and normalize both baseline and efficient energy use to typical

operations throughout a year.

In general, our analysis will be based on conditions as they were observed, and our analysis will

be applied to the first year after measures became operational. If we find equipment that is

temporarily out of service for maintenance or some other reason, we may apply adjustments to

first-year extrapolations. We also reserve the right to assess the most appropriate long-term

conditions on which to base measure lifetime savings. Our data collection process will identify

when and why operations changed, and we will document any changes that are expected in the

future. This will enable us to base our savings analysis on an average long-term condition or on

a condition that is expected to remain stable over time.

In some cases, particularly for large industrial gas processing facilities, project savings may

represent a significant portion of overall billed usage on a meter. In those cases, we may request

the site’s billing data, so we can determine savings in accordance with IPMVP Option C. While

interval data is preferred for billing-based analyses, we recognize that it is not always available,

particularly for gas projects. If metered billing data is weather sensitive, we will weather

normalize it. Similarly, we will normalize billed usage to production to adjust for any variations

in operations. While billing analysis is not the most common method for determining savings at

industrial sites, it can be appropriate for large gas projects, particularly in unheated facilities.

Similarly, in cases where there are multiple electric meters at a site and the project has affected

the load on one meter significantly, such as at some chiller plants, normalized billed usage can

be the most accurate reflection of savings.

After each correction type is applied, we will record the new gross savings estimates and

associate these estimates with each correction type in our evaluation database. For each

evaluated project, we will summarize our findings from the interviews and site visits in a

project-specific M&V report. These site reports will be “sanitized,” in that they will not contain

information that could identify the customer, except for PA-assigned and PA-trackable

identification numbers.



5.3 Net Savings Evaluation

Our approach to estimating net savings is to estimate a net-to-gross ratio (NTGR) that will be

multiplied by gross savings to obtain net savings. Our estimation of the NTGR to support the

estimation of net savings relies on these documents:

� TechMarket Works. 2006. California Energy Efficiency Evaluation Protocols: Technical,

Methodological and Reporting Requirements for Evaluation Professionals {a.k.a. Evaluators’

Protocols}. San Francisco, CA: CPUC.

� Ridge, Richard, Ken Keating, Lori Megdal, and Nick Hall. 2007. Guidelines for Estimating

Net-To-Gross Ratios Using the Self Report Approach. Prepared for the California Public Utilities

Commission.

� CPUC. 2015. Methodological Framework for Using the Self- Report Approach to Estimating Net-to-

Gross Ratios for Nonresidential Customers. San Francisco, CA: CPUC.

� Itron. 2015. 2013 Custom Impact Evaluation Industrial, Agricultural, and Large Commercial. San

Francisco, CA: CPUC.

� Itron. 2017. 2015 Custom Impact Evaluation Industrial, Agricultural, and Large Commercial. San


� TechMarket Works. 2006. The California Evaluation Framework. San Francisco, CA: CPUC.

� Ridge, Richard, Phillipus Willems, Jennifer Fagan and Katherine Randazzo. The Origins of

the Misunderstood and Occasionally Maligned Self-Report Approach to Estimating the

Net-To-Gross Ratio. Presented at the International Energy Program Evaluation Conference

in August 2009.

5.3.1 Questionnaire Development

To ensure consistency between this evaluation effort and former custom evaluations, we will

base our development of the new 2018 and 2019 NTGR survey instruments on the

questionnaires designed for the evaluation of the 2015 and 2017 Custom programs. As we

review those former instruments we will draw upon pertinent lessons learned from those

evaluations, as well as drawing upon other recently completed evaluations and methodological

reviews by PWP, Evergreen Economics, Tetra Tech, NMR Group, and DNV-GL. We will also

coordinate closely with DNV-GL and the rest of the Group A evaluation team to ensure

consistency between our current evaluation efforts.

In all, we will develop three questionnaires, one for each level of rigor: basic, standard, and

enhanced.

To assess the participants who were assigned to the basic and standard levels of rigor, we will

use a computer-assisted telephone interview (CATI) survey, as was done for the evaluation of

the 2015 Custom program. To further improve consistency with the Group A evaluation effort,



we will field the CATI surveys using the same firm that they are working with. The basic survey

will take approximately 20 minutes to complete, while the standard survey will take

approximately 30 minutes.

For participants assigned to the enhanced level of rigor, personal telephone interviews will be

conducted by our staff of highly-trained professionals whose knowledge and experience are

commensurate with the interview requirements. Due to the complex nature of these projects,

our interviewers will also be assisted by an experienced engineer. Potentially, these interviews

will involve conversations with several people who were involved in the project, including the

primary decision-maker, operations staff, vendor representatives, program staff, and other

decision influencers.

Once the draft versions of each questionnaire are ready, we will pre-test each of them on two

participants apiece to ensure that the survey length is not onerous, the wording is clear, the

question sequence is appropriate, and skip patterns and consistency checks are functioning

correctly.

5.3.2 Data Collection

To achieve the highest possible response rate, we will use several strategies. First, the team

responsible for estimating gross savings will work with site personnel (referred to as operations

staff) to determine whether they or someone else was the key decision maker. If they are the

key decision maker, the team will inform them that they will be contacted by a representative of

Opinion Dynamics to schedule the NTGR interview. They will also underscore the importance

of participating in this interview and that their answers will be kept strictly confidential.

If they are not the key decision maker, the team will proceed to conduct a brief interview with

the operations staff to collect additional information that will be used in the analysis of program

influence as well as to obtain the name and contact information (e-mail address, US Postal

address and telephone number) of the key decision-maker who will then be contacted by

Opinion Dynamics to schedule the NTGR interview.

Second, we will send each sampled participant, identified as the key decision maker, an e-mail

that: 1) explains that the CPUC is sponsoring this evaluation of the Custom program, 2)

underscores the importance of this evaluation and the confidentiality of their responses, and 3)

provides the names and telephone numbers of IOU program managers and key CPUC staff who

can verify the legitimacy of the survey. The e-mail will also include a list of the topics and

questions covered in the survey so that the customer can prepare in advance. In cases where an

e-mail address is not available, we will send a letter containing the same information via US

mail.

We anticipate that we will not obtain completed NTGR surveys for all 235 participants in the

gross savings sample. The response rate for the 2015 Custom program NTGR survey was 75%.

So, to complete a total of 235 NTGR surveys and to complete the additional 65 sites needed to



reach our goal of 300 completed NTGR surveys, we will increase the total number of sampled

sites by approximately 100.

When the surveys are complete, we will provide the CPUC with a detailed sample disposition

table that reports response rates in a manner consistent with the standards developed by the

American Association for Public Opinion Research. We will report other metrics and survey

outcomes as requested by Energy Division staff.

5.3.3 NTGR Analysis

Once data has been collected and cleaned, we will estimate the life-cycle-savings-weighted

NTGR for each domain using the approach described in section 6.2. We will then conduct

analyses of NTGR reliability, NTGR sensitivity, and NTGR drivers.

5.3.3.1 NTGR Reliability Analysis

We will use Cronbach’s alpha19 to assess and report the reliability of our estimated NTGRs to

ensure that our measure of program influence is internally consistent. Cronbach’s alpha is a

widely accepted measurement of the internal consistency and reliability of a multivariate

measurement that can be used to measure an underlying construct such as program influence.

Alpha represents the reliability of an overall set of measurements such as program attribution

index scores. Generally, alpha should be 0.70 or above. While Cronbach’s alpha increases as the

number of items in the scale increases, we usually have only three to four items distilled from

multiple questions. So even with consistency checks, inconsistencies invariably emerge, leading

to low inter-item correlations. As a result, an alpha lower than the normal threshold of 0.70

might be acceptable. For large energy savers, we will consider additional quantitative and

qualitative data that is not reflected in the alpha. When at least two raters are involved in

independently estimating the NTGR, we will also calculate inter-rater reliability.

5.3.3.2 NTGR Sensitivity Analysis

To assess the stability and possible bias of the estimated NTGR, we will conduct sensitivity

analyses, which may involve changing weights and changing the questions used in estimating

the NTGR.

5.3.3.3 Assessing the Drivers of the NTGR

Behind the net-to-gross ratios calculated for each project are several contextual factors that

might have influenced the project, either directly or indirectly. To assess the drivers of the

NTGRs, we will employ a two-pronged approach that combines a quantitative regression

analysis with a qualitative examination of the factors that influenced customer decision-makers.

19 Cronbach L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.



Together, these analyses will enable us to make actionable recommendations to improve

program influence.

Quantitative Approach

We will use a regression analysis to examine the program, non-program and participant

variables that might be associated with high free-ridership (low NTGR) or low free-ridership

(high NTGR). The intent of this analysis will be to look beyond the numerical responses used in

the NTGR algorithm to better understand the factors that influenced the decision-makers on the

project. The final file will include analysis of variables such as average monthly electricity bill,

importance of the rebate, importance of the technical assistance, previous program experience,

square footage, presence of a company policy regarding the purchase of energy-efficient

equipment, vendor influence, timing of the program’s involvement and at what point in their

decision-making process the customer became aware of the program, the number measures

installed, the end use affected by the new equipment, and , and the level of NTGR rigor they

received.

We will specify a variety of regression models, and, for each model, regression diagnostics

including influential observations and collinearity will be reported followed by any will undergo

any necessary adjustments. We will also calculate a variety of weights for use in the analyses.

For example, the effect of a corporate energy policy that is rigorously enforced might be

negative while the effect of the rebate on reducing the simple payback period might be positive.

The general form of the models is this equation:

�� = � + ∑ � + �� + �� where:

�� = NTGR for the ith customer

� = the intercept

�� = a vector of customer and program/non-program characteristics for the ith customer

� = a vector of k coefficients that reflect the change in the NTGR associated with a one-unit change in the kth explanatory variable

�� = captures the differences in NTGRs among the various customers that are not explained by the model

This regression analysis will lead to an improved understanding of the factors underlying the

survey results. It will also help us to better understand program influence by highlighting the

characteristics of the strongest and weakest groups of projects. These insights will in turn help us

to prepare a set of actionable recommendations to improve program influence.

Qualitative Approach

As with the quantitative analysis, the goal of qualitative analysis will be to highlight the factors

(both program and non-program) and characteristics of the groups of projects with both the



strongest and weakest levels of program influence. As such, we will examine the same set of key

contextual factors that we considered in the regression analysis. This qualitative analysis will be

performed for the lowest and highest NTGR ratio quartiles, i.e., the group with the lowest

NTGRs and the group with the highest NTGRs. As with the quantitative analysis, the insights

we gain from this effort will help us to prepare actionable recommendations to reduce free

ridership.

5.4 Cost, Lifetime, and Load Shape

This section describes our approach to evaluating the measure costs, lifetimes, and load shapes

that we will use to estimate cost-effectiveness, as described in section 6.3.

� Incremental measure cost. We will evaluate measure costs by reviewing project

documentation to verify that valid and appropriate costs were included per the rules defined

in the 2018 Statewide Customized Offering Procedures Manual for Business20. For add-on

measures, we will consider full measure cost. For early-retirement measures, we will confirm

that the early-retirement cost was correctly calculated as the total installation cost minus the

net present value of the total cost that would have been incurred to install an ISP measure at

the end of the remaining useful life period. For replace on burnout measures and new

construction measures, we will verify the marginal cost of implementing the measure by

comparing the incremental measure cost to the measure cost of equipment that meets code

or industry standards.

� EUL and RUL. We will use all the available information for each claim to assign it the most

appropriate EUL from DEER. If we determine that the claim is an early-retirement

measure, we will assign it a RUL equal to one-third of the EUL, unless the PA has

substantiated an alternate RUL.

� Load shape. For each of the claims comprising the sampled project, we will select the most

appropriate load shape from the load shape library in the CPUC’s cost effectiveness tool.

20 PG&E, SCE, SoCalGas, SDG&E. 2018. 2018 Statewide Customized Offering Procedures Manual for Business. San




6 Portfolio Savings and Cost-Effectiveness

We will estimate gross savings, net savings, and cost-effectiveness for the portfolio of custom

projects claimed in 2018 and 2019. We will prepare separate estimates for each of the domains

in the sample design and for all projects associated with each PA.

6.1 Gross Savings

We will calculate the domain-level ex post gross savings and the achieved precision for each fuel

type using the stratified mean estimation method and the stratified ratio estimation method21.

For the stratified mean estimation method, the steps are as follows:

1. Calculate the mean ex post gross savings

�̄st=��̄��

h=1

where:

Wh = N

N

h = which is the stratum weight

Nh = population of stratum h

N = population of domain

y h = the mean of y for stratum h

y st = the mean resulting from a stratified random sample (st for stratum).

2. Calculate the domain ex post gross savings

� = �� The realization rate is the ratio of the domain ex post gross savings to the domain ex ante

gross savings:

�� = YX

where:

X = the sum of the ex ante savings in the domain

3. Calculate the variance of the mean, S2

21 Cochran, William G. (1977). Sampling techniques. New York: John Wiley & Sons.



∑∑=

L

1=h

2

hhL

1h h

2

h

2

hst

2 sW -

n

s W = )y(

NS

where:

2

hs = the stratum variance

nh = the stratum sample size

The second term in the equation represents the finite population correction.

The achieved relative precision at 90% confidence is:

�" = 1.645'(��)��

For the stratified ratio estimation method, the steps are as follows:

1. Calculate the ratio (realization rate), b

∑

∑

=

==n

i

ii

n

i

ii

xw

yw

b

1

1

where:

wi = Nh/nh

yi = ex post savings

xi = ex ante savings for sample point i

Large energy savers, selected with certainty, will be assigned a sample weight of 1. Other

projects, selected at random, represent more than one project in the population and thus will

be assigned a weight greater than 1.22

2. Calculate the residual of each sampled point, ei = yi−bxi, then the standard error across

strata for each domain is:

( )( )

∑

∑

=

=

−

=n

i

ii

n

i

iii

xw

eww

bse

1

1

21

22 Savings for projects in the exclusion stratum will be passed through, i.e., y7i=xi.



and the achieved relative precision at 90% confidence is:

( )b

bserp

645.1=

We will also apply the steps above to derive PA-level ex post estimates for electric and gas

savings. We will determine which method, stratified mean estimation or stratified ratio

estimation, gives the most reliable estimate of precision for the PA-level realization rate.

In addition to reporting the domain ex post gross savings and achieved precision, we will also

provide a table of the sample disposition that will include the response rate and the number of

replacements used in each domain. We will report any potential for non-response bias, and, if

necessary, weight the results to correct for any bias.

6.2 Net Savings

Within each of the sample domains that we established for estimating the gross savings, we will

estimate the ex post net savings for the population in each domain by multiplying the ex post

gross savings for the population by the sample-based ex post life-cycle savings-weighted NTGR.

The resulting ex post net savings will then be summed up across the domains related to each PA

to calculate the total ex post net savings along with the achieved relative precision. We will then

sum the PA ex post net life-cycle savings across the five domains to arrive at the estimate for

total statewide portfolio savings. First, for the two parameters, the ex post gross savings (GS)

and the NTGR, the 90% relative precision (RP) is calculated. Next, the 90% relative precision of

the net savings for each domain (D) is calculated in a way that, using the equation below,

accounts for the propagation of errors involved in multiplying these two parameters.23

2

D

2 )rp(NTGR )(SavingsNet Domain += DGSrpRP

where:

)( DGSrp = the 90% relative precision of the ex post gross savings

)( DNTGRrp = the 90% relative precision of the ex post NTGR

The error bound of the domain net savings will then be calculated using this equation:

SavingsNet DomainRPNSEB DD ×=

23 TecMarket Works, Megdal & Associates, Architectural Energy Corporation, RLW Analytics, Resource Insight, B &

B Resources, Ken Keating and Associates, Ed Vine and Associates, American Council for an Energy Efficient Economy, Ralph Prahl and Associates, and Innovologie. (2004). The California Evaluation Framework. Prepared for the

California Public Utilities Commission and the Project Advisory Group; Taylor, John R. (1997). An Introduction to

Error Analysis: The Studies of Uncertainties in Physical Measurements. Sausalito, CA: University Science Books.



where:

DEB = the 90% error bound of net savings for a domain

DNS = the net ex post savings for a domain

Once the error bound of net savings for each of the domains is calculated, the error bound for

each PA and the statewide portfolio can be calculated using this equation:

) Domain(EB .....) Domain(EB ) DomainEB( 2

n

2

2

2

1 ++=PortfolioEB

The equation for EBD above is based on these assumptions:

1. There are no interactions between the domains,

2. Each of the individual domains has been evaluated independently, and

3. Each evaluation has provided an unbiased estimate of the actual savings of the

corresponding domain

The result is a simple consequence of (a) the fact that the standard deviation of a sum of

statistically independent random variables (e.g., the estimated savings of each program) is the

square root of the sum of the squares of the standard deviations of each of the random variables,

and (b) the error bound being defined as 1.645 times the standard deviation.

Finally, the relative precision of the portfolio for each PA and the statewide portfolio, defined as

a combination of two or more domains, will be calculated using this equation:

Portfolio

Portfolio

PortfolioNS

EBRP =

where:

PortfolioEB = the error band of the PA portfolio or statewide portfolio.

(Only the net savings estimated via ex post EM&V are included in

PortfolioEB . It excludes net savings from measures whose parameters

were not evaluated, were passed through, or were DEER based. It also excludes interaction effects.)

PortfolioNS = the net savings impacts (kWh, kW or therm) of the PA portfolio or

statewide portfolio. ( PortfolioNS includes all net savings. That is, the

net savings estimated via ex post EM&V, as well as the net savings from measures whose parameters were not evaluated, were passed through, or were DEER based. It also includes interaction effects.)

The expected relative precision will improve anytime two or more domains are combined for

reporting purposes due to the increase in the number of observations. The best level of relative

precision will be for the portfolio that includes all the domains. Results will also be reported by

electric and gas separately for the same combinations.



6.3 Cost-Effectiveness

We will use the CPUC cost-effectiveness tool (CET) to compute life-cycle costs and benefits for

each claim, using our ex post estimates of net savings by baseline, EUL/RUL, load shape, and

incremental costs. These cost and benefits will be aggregated to the domain level and PA level

using methods described above and will then be used to compute cost-effectiveness ratios such

as total resource cost (TRC).



7 Data Management and Quality Control

We will develop an Excel workbook to standardize how we document all primary data

collection and modelling for each project and for the constituent claims. The workbook will

have a series of locked tabs that contain data-entry ranges and that validate each value entered.

Instructions for entering data will appear in adjacent cells. Our analysts will add additional tabs

to the workbook to document the models and inputs used to calculate savings.

Our senior engineering staff will:

� Develop the workbook

� Train our project-review staff

� Provide quality control

� Resolve problems

In their quality-control role, our senior engineering staff will review each of the complete

project-evaluation workbooks and interact as necessary with our analysts to ensure that all

projects are treated consistently.

Our data-processing team will compile data for the project workbooks and associate that data

with claim data. They will apply additional quality-control tests to the data and refer any issues

that need resolution to the assigned analyst.



8 Task Plan

This section describes how we will complete the ex post evaluation of 2018 and 2019 custom

projects. Its structure corresponds to the budget and schedule portions of the workplan.

8.1 Workplans and Updates

This document is our initial IALC workplan. We will submit an updated IALC workplan in

early 2020 to refine and improve our approach, and to respond to changes as programs evolve.

8.2 Data Collection and Sampling Plans

8.2.1 Sample Frame

We will obtain data from CEDARS on all PA claims for 2018 and collaborate with the Group

A evaluation team to determine how the custom claims will be segregated from all other claims.

This will be done as soon as the PAs confirm the 2018 claim has been finalized. Similarly, we

will obtain and identify custom claims for each quarter of 2019.

8.2.2 Review of Reported Custom Claims

We will review the 2018 claims to identify whether additional information is required to support

a review of compliance with CPUC policies. We will issue requests for any necessary additional

information that is identified during this review. Once we have all the requested data, we will

complete the review as described in section 3.2. We will repeat this process for the 2019 claim.

8.2.3 Sample Selection

We will apply the 2018 sample design to the sample frame and select the ex post evaluation

sample, including enough projects to satisfy the sample-size objectives for gross- and net-savings

evaluations. We will apply a similar design to select samples representing each quarter of 2019,

with the final quarter’s sample selected after the PAs confirm that the 2019 claim is final.

8.2.4 Data Requests Related to Sampled Custom Projects

We will request that PAs upload all files for the 2018 sampled projects. We will review these

files and then request that PAs provide any missing critical data, models, or documentation. We

will repeat this process for each of the 2019 quarterly project samples.



8.2.5 Site Data Collection

For each project in the 2018 and 2019 gross savings samples, we will develop an M&V plan that

will serve as a roadmap to guide the customer interviews, site visits, data collection, and the

procedures to estimate savings. Each M&V plan will be customized to the site conditions, the

type of measures installed, and the assigned rigor level for gross savings and NTGR estimation.

We will collect all the data specified by each M&V plan, including the data obtained by

completing the NTGR survey. We will also expand the number of projects in the sample for

each of the 2018 and 2019 NTGR surveys to ensure that we obtain a sufficient number of

completed surveys for the analysis.

8.3 Gross Savings Estimates

For each project in the 2018 and 2019 gross sample we will process and prepare an analysis-

ready version of the data collected. We will finalize the M&V procedures to reflect the observed

conditions for each project. We will then estimate gross savings for each claim comprising each

of these projects. For claims involving normal replacement measures, we will estimate savings

for the first baseline period. For early-retirement measures, we will estimate savings for both the

first and second baselines. We will use the results from the sample to estimate gross life-cycle

savings for each sample domain and for each PA.

In addition, for each of the claims comprising the sampled projects, we will estimate

incremental measure costs and assign the appropriate measure lifetime and load shapes.

8.4 Net Savings Estimates

We will pre-test the NTGR questionnaires and make any necessary revisions. We will then

complete the NTGR with 2018 and 2019 net savings samples. We will process the data from

each survey and prepare an analysis-ready version of the data, which will then be used to

estimate NTGR for each of the claims comprising these projects. We will use the results from

the sample to estimate net life-cycle savings for each sample domain and for each PA.

Using the NTGR estimates for each of the claims, we will estimate life-cycle costs and benefits,

and then use the sample to estimate cost-effectiveness results for each sample domain and PA.

8.5 Final Reports

We will develop a draft and final evaluation report for 2018 and 2019. These reports will

document our evaluation methodology and present our estimates of gross and net savings and

cost-effectiveness. These reports will include an analysis of the reasons for corrections to the

reported claims, including separate estimates of the change in savings due to our review of the

reported claims (see section 3). The reports will also include actionable recommendations



designed to improve the PA’s savings claims for future custom projects and to improve future ex

post evaluation methods.

8.6 Data Documentation

We will document all our primary data collection, modelling, and data processing procedures so

that our results are transparent and can be reproduced by other parties.



9 Budgets and Schedule

This section presents our budget and schedule for evaluating the 2018 and 2019 IALC

programs.

9.1 Evaluation Budget

The IALC evaluation is part of the larger budget for all early feedback and ex post evaluation

(Group D contract deliverables 9, 10, 13, and 17). The other portion of this larger budget is for

the evaluation of strategic energy management (SEM) projects. Table 8 presents our budget in

total and separately for SEM and for the IALC activities discussed in this workplan.

Table 8: IALC Evaluation Budget

Budget by category ($)

Deliverables IALC SEM Early feedback and ex post evaluation (Total

of IALC and SEM)

Workplans and updates 168,000 168,000 336,000

Data collection and sampling plans 129,000 43,000 172,000

SEM early feedback — 495,000 495,000

Gross savings estimates 3,632,000 353,000 3,985,000

Net savings estimates 640,000 60,000 700,000

Final reports 343,000 171,000 514,000

Data documentation 133,000 67,000 200,000

Total 5,045,000 1,357,000 6,402,000

9.2 Timeline and Milestones

Figure 2 presents our timeline and milestones.



Figure 2: Timeline and Milestones

Year:

Deliverable Task / Subtask Due date Qtr: 1 2 3 4 1 2 3 4 1 2 3 4

Evaluation work plan . 2

Draft release - ED review 20-May-19 2

Draft release - stakeholder review 7-Jun-19 2

Final posted 6-Jul-19 2

Plan update Apr-20 2

Data collection and sampling plan Jul-19 . 2

2018 project data requests Jun-19 2

2019 project data requests Jul-19 2 2 2

Gross savings estimates



Net savings estimates



Final evaluation report

2018





2019





Data transfer and documentation

2018 Apr-20 2

2019 Apr-21 2

Key:

Activity to complete deliverable expected

Major deliverable to be completed

2019 2020 2021



10 Management and Staffing Plan

Figure 3 shows an organizational chart of our management and staffing structure.

Figure 3: IALC Evaluation Organizational Chart

Mike Baker

Principal in Charge

IALC Gross Savings Team

Jeff Romberger

Josh Weissert

Mike Yim

Engineering Staff

IALC Net Savings Team

Rick Ridge

Katherine Randazzo

Survey Staff

IALC Data Team

Faith DeBolt

Gina Hicks

Technical Staff

Chris Smith

IALC Evaluation Lead

Workplan for 2018-19 IALC Custom Impact Evaluation · Total 2,449 140 51% 55% 5% 229 95 82% 5% ....

Documents

Transcript of Workplan for 2018-19 IALC Custom Impact Evaluation · Total 2,449 140 51% 55% 5% 229 95 82% 5% ....