MAPE Publication

Neil McAndrews

For Bob Ryan of Deutsche Bank

What makes a forecast a valid forecast?

Unbiased – overshoots as many time as it undershoots

Validation Metric - Percent Error

Accurate – predicts the actual loadValidation Metric – Mean Absolute Percentage Error

Reliable – often predicts the load accuratelyValidation Metric – histogram of error

1. 2.

Update of forecast performanceSept 1 to Dec 3. ERCOT Peak Forecast and Actual (from PUC data)1. MAPE 4.01%2. Average Bias is 1010 MWs.

-15.00%

-10.00%

-5.00%

0.00%

5.00%

10.00%

15.00%

20.00%9/

1/20

06

9/8/

2006

9/15

/200

6

9/22

/200

6

9/29

/200

6

10/6

/200

6

10/1

3/20

06

10/2

0/20

06

10/2

7/20

06

11/3

/200

6

11/1

0/20

06

11/1

7/20

06

11/2

4/20

06

12/1

/200

6

Last Month's PRS Data Updated

November 30th PUC data

Peak Load Forecast: 40,627 MW Actual Peak Load: 47,000 MW Forecast Error: 6,573 MW Percent Forecast Error: minus 13.6%

Month

Mean Absolute Percent Error for MTLF run at 16:00 (2004)

Mean Absolute Percent Error for MTLF run at 16:00 Day Ahead (2005)

Mean Absolute Percent Error for MTLF run at 16:00 (2006)

January 4.76 5.02 4.79

February 4.81 3.47 4.81

March 2.89 3.41 4.95

April 3.23 3.37 5.59

May 3.93 4.21 3.59*

June 4.85 3.91 2.65*

July 5.07 5.22 2.62*

August 4.10 4.86 2.09*

September 3.66 6.24 4.03

October 3.85 4.51 3.63

November 3.60 4.53 3.28

December 4.18 5.95 NAAverage MAPE for year

4.08 4.56 4.44

* Don't include 1800 MW bias

ERCOTs Monthly MAPE for 16:00 MTLF

Raises questions regarding which model is being reported. Is it the AREVA model alone, or a combination of models, and why it doesn’t include this summer’s bias?

ERCOT MAPES

Each model should have its percent error and MAPE measured Actual operational forecasts (the combination of model

outcomes) should be also be measured because the select process can increase error and introduce a bias into the process, e.g. selecting the larger of two forecasts.

For example: instead of using the current two forecasts methods introduce a third forecast that has a peak of 65,000 MWs. This third forecast method would always be selected. But this would be wrong because it would also introduce bias and have a large error.

Forecasts should be used based upon highest reliability, lowest MAPE and lack of bias.

Summary of Current Forecast

The forecast is overestimating peak load for the months of September through December 3 by an average of 1,010 MWs (2.12%). Actual error should be much closer to zero. (Overestimation does appear to be declining later in the period)

In November the bias appears to be negligible.

Suggest a review of this data for accuracy.

Summary

MAPE is a measurement tool that improves fundamental market information for market participants.

Market participants and the ISO can continually iterate on their forecasts to improve forecast accuracy (i.e., minimize forecast error). This is common practice in other regions, e.g., PJM.

Publishing load and weather MAPEs will help to lower forecast error by the natural collaboration between ERCOT and market participants.

ERCOT has not isolated the weather forecast error from the overall model. This PRR addition would isolate the causes of errors, and lead to improved forecast performance for each of the modeling method as well as a combination of forecast outcomes.