Baseline AnalysisCBP, AMP, and DBP

Steve Braithwait, Dan Hansen, and Dave Armstrong

Christensen Associates Energy Consulting

DRMEC Spring Workshop

May 7, 2014

May 2014 1

May 2014 2

Presentation Outline

Objectives Methodology

Data Performance measures

Aggregator program (CBP and AMP) results Demand Bidding Program (DBP) results

May 2014 3

Objective: Assess Performance of Alternative Baseline Types

For each Utility and Notice type: All customers, with BL adjustment as chosen All customers, simulated with universal selection

of the BL adjustment Sum of individual BL vs. portfolio BL (constructed

from aggregated customer loads), for AMP and CBP only

Examine unadjusted and day-of adjustments with 20%, 30%, 40%, 50% caps, and uncapped

May 2014 4

Analysis Details

For actual program event days The “true” baseline is the estimated reference

load from the ex post evaluation For event-like non-event days

The “true” baseline is the observed load

May 2014 5

Performance Measures (1)Percentage Baseline Error

Percentage BL error for each customer/portfolio-event day is: Percentage error = (LP

d – LAd) / LA

d

LAd = actual, or “true” baseline load on day d

LPd = “predicted” baseline to be evaluated

Positive value = over-estimated baseline (implies over-stated program load impact)

Negative value = under-estimated baseline (implies under-stated program load impact)

May 2014 6

Performance Measures (2) Accuracy

Accuracy is measured as the median absolute percentage error (MAPE) Calculate the absolute value of the percentage error for

each customer/event-day Calculate the median of values across customer/event-

days (mean can be misleading due to extreme values) Higher values correspond to larger baseline errors

May 2014 7

Performance Measures (3)Bias

Bias is measured by the median percentage error, without taking the absolute value

Positive values indicate upward bias (i.e., the program baseline tends to over-state the “true” baseline)

Negative values indicate downward bias (i.e., the program baseline tends to under-state the “true” baseline)

Nominated Customers by Choice of BL Adjustment – CBP and AMP

May 2014 8

May 2014 9

Accuracy (Median Abs. % Error)PG&E CBP-DO

May 2014 10

Bias (Median % Error)PG&E CBP-DO

May 2014 11

Percentiles of % Errors – PG&E CBP-DOActual Events, by Adjustment Cap

May 2014 12

Percentiles of % Errors – PG&E CBP-DOSimulated Events, by Adjustment Cap

Summary: Accuracy & Bias (Aggregated Indiv.; Universal Adj.; 40% cap)

May 2014 13

Summary: Percentiles of % Errors(Aggregated Indiv.; Universal Adj.; 40% cap)

May 2014 14

May 2014 15

Summary of Findings Accuracy and bias measures vary by utility, program

and notice type Suggests that factors other than baseline type and

adjustment caps may be most important, such as types of customers (e.g., highly variable load) and event-day characteristics (e.g., event on isolated hot day)

Day-of adjustment often improves accuracy and reduces bias, but level of cap is less important Largest errors typically occur for Unadjusted BL and

Unlimited cap BL with small median error (e.g., 1%) can have >10%

errors in 20 percent of cases

16May 2014

DBP Results:PG&E Distribution of % Errors

-100.0%

-90.0%

-80.0%

-70.0%

-60.0%

-50.0%

-40.0%

-30.0%

-20.0%

-10.0%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Unadj. 20% 30% 40% 50% No Cap

% B

as

eli

ne

Err

or

Baseline Method

5th %ile

10th %ile

25th %ile

Median

75th %ile

90th %ile

95th %ile

17May 2014

DBP Results:SCE Distribution of % Errors

-40.0%

-30.0%

-20.0%

-10.0%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

Unadj. 20% 30% 40% 50% No Cap

% B

ase

line

Err

or

Baseline Method

5th %ile

10th %ile

25th %ile

Median

75th %ile

90th %ile

95th %ile

Summary

Day-of adjustments tend to improve baseline accuracy and reduce bias

The analysis provides support for making the day-of adjustment the default option

The effectiveness of the day-of adjustment is not very sensitive to the level of the cap

May 2014 18

May 2014 19

Questions?

Contact – Steve Braithwait or Dan Hansen, Christensen Associates Energy ConsultingMadison, Wisconsin Steve@CAEnergy.com Danh@CAEnergy.com 608-231-2266

Appendix

SCE – CBP DO SDG&E – CBP DO PG&E – AMP DO SCE – AMP DO

May 2014 20

May 2014 21

Accuracy (Median Abs. % Error)SCE CBP-DO

May 2014 22

Bias (Median % Error)SCE CBP-DO

May 2014 23

Percentiles of % Errors – SCE CBP-DOActual Events, by Adjustment Cap

May 2014 24

Percentiles of % Errors – SCE CBP-DOSimulated Events, by Adjustment Cap

May 2014 25

Accuracy (Median Abs. % Error)SDG&E CBP-DO

Accuracy – Med. Abs. Err. (MW)SDG&E CBP DO

May 2014 26

May 2014 27

Bias (Median % Error)SDG&E CBP-DO

May 2014 28

Accuracy (Median Abs. % Error)PG&E AMP-DO

May 2014 29

Bias (Median % Error)PG&E AMP-DO

May 2014 30

Accuracy (Median Abs. % Error)SCE AMP-DO

May 2014 31

Bias (Median % Error)SCE AMP-DO