FANCCO 2015

Uncertainty Handling Using Neural Network-Based Prediction Intervals

A/Prof Dipti SrinivasanDepartment of Electrical & Computer Engineering

National University of Singapore

Outline

Introduction – Uncertainty modeling methods

NN-based PIs for Forecast Uncertainty Modeling

Advantages of PIs compared with point forecasting

LUBE Method for Constructing NN-based Prediction

Intervals

Case Studies:

Testing on data sets

Uncertainty Handling in Smart Grids: Eectrical Load

and Wind Power Forecasting

Conclusions

Probabilistic Methods• Probability density function (PDF) & cumulative distribution function (CDF)

Stochastic models• Stochastic programming• Monte Carlo simulation

Prediction Intervals• the upper bound, lower bound and the coverage probability.

Fuzzy Logic

A fuzzy logic system and its components

Uncertainty Modeling Methods

Forecasting & Estimation Methods

Forecasting methods

Point Forecasts

Interval Forecasts

Confidence Intervals

Boot Strapping

PredictionIntervals

• Point forecasts cannot properly handle the uncertainties associated with data sets.

• PIs are excellent tools for the quantification of uncertainties associated with point forecasts and predictions

Prediction Intervals:Most forecasters do realize the importance of providing interval forecasts to enable users to• Assess future uncertainty,• Plan different strategies for the range of possible outcomes

indicated by the interval forecast,• Compare forecasts from different methods more thoroughly,

and• Explore different scenarios based on different assumptions

more carefully.

Confidence intervals: are intervals constructed about the predicted value of y, at a given level of x, which are used to measure the accuracy of the mean response of all the individuals in the population.

Prediction intervals: are intervals constructed about the predicted value of y that are used to measure the accuracy of a single individual’s predicted value.

Confidence Intervals v/s Prediction Intervals:

Artificial Neural Networks • Feedforward Neural Network using back propagation with momentum

learning and adaptive learning rate are often used for forecasting and prediction applications

• The learning rule modifies the weights according to the input patterns that it is presented with.

• In a sense, NNs learn by example as do their biological counterparts.• Neural Network is a powerful tool for non-linear mapping

Input

Des iredOutput

• Prediction intervals (PIs), have been proposed to model uncertainty in forecasting studies.

• Advantages of PIs over point forecasting:• When uncertainty exists in the data, such as multi-valued,

sparse, noisy datasets or if targets are affected by probabilistic events, the reliability of point forecasts significantly drops.

• NN point predictions: only provides predicted values but no information about prediction accuracy;

• PIs not only provide a range that targets are highly likely to lie within, but also have an indication of their accuracy (confidence level)

PIs v/s NN-based Point Forecasting

Construction of Neural Network-Based Prediction IntervalsMain motivation for construction of PIs is to quantify the likely uncertainty in the point forecasts• Delta Method - The root theory of delta method is nonlinear regression• Bayseian method – NNs are trained based on a regularized cost

function• Mean-variance estimation method – assumes that errors are normally

distributed around the true mean of targets and estimates the target variance using a NN

• Bootstrap Method – Most common; uses an ensemble of NN models to produce a less biased estimate of the true regression of the targets

Construction of Neural Network-Based Prediction IntervalsTraditional methods construct PIs in two steps:

1) They regress the given dataset to a specified model or function (which is the same as point forecasts)

2) According to the assumed data distribution, the statistical mean and variance values are calculated• if Jacobian or Hessian matrix are needed, they are also

calculated at this step. Based on this information, PIs are then constructed.

• Implementation of these methods is complex. – For example, Delta and Bayesian methods need to calculate the Jacobian matrix and Hessian matrix of the parameters in each iteration.

• Traditional methods make assumptions about the data distribution. –Delta method assumes that the noises are normally distributed and t-distribution is applied–Mean-variance estimation method assumes that NN (predicting the mean) can precisely estimate the true mean of the targets. –Bootstrap method assumes that an ensemble of NN models will produce a less biased estimate of the true regression of the targets.

• Massive computational requirements hinder widespread applications of these methods for decision-making

Construction of NN-Based Prediction Interval Methods: Disadvantages of traditional methods

Construction of NN-Based PIs for Uncertainty Modeling using LUBE method

A new method:• Lower Upper Bound Estimation (LUBE)• NN with two outputs to directly generate the upper and lower bounds.• Makes no assumptions about the dataset• Simpler and avoids calculation of derivatives of NN output with

respect to its parameters• Much smaller computational requirements

Prediction Interval Coverage Probability (PICP)

Prediction Interval Normalized Average Width (PINAW)

Prediction Interval Normalized Root-mean-square Width (PINRW)

Objective Function

PI Evaluation Indices

Particle Swarm Optimization (PSO)

Flocks of birds → Particle Swarm Optimization (PSO)

Takes inspiration from Natural swarms for solving optimization problems

Transforms the swarm intelligence metaphor into engineering methodologies to solve optimization and search problems.

Powerful Optimizer- Used for optimizing the structure of Neural Network for construction of Pis in this work

Construction of NN-Based PIs for Uncertainty Modeling

Determination of the Optimal NN

structure using particle swarm

optimization

Velocity and Position Update

Mutation Operator

PI Construction and Evaluation for

Training Set

Update pbest and gbest particles

Testing and Evaluation

Flow chart of PSO-based LUBE method

Construction of NN-Based PIs for Forecast Uncertainty Modeling

Datasets for case studies

1. Ding10 is a one-dimensional synthetic mathematical function, with the added noise with a heterogeneous distribution.2. HAS is a five-dimensional synthetic mathematical function. Unlike case study 1, the added noise is normally distributed with a constant variance.3. Dry bulb temperature (DBT) comes from an industrial dryer sampled every ten seconds. Three inputs are used for estimating the output of dry bulb temperature.4. Data in case study 4 comes from a medical study, which contains 315 observations on 14 variables. This study tries to investigate the relationship between personal characteristics, dietary factors, and plasma beta-carotene.5. T70 comes from a real baggage handling system, which is frequently affected by probabilistic events. The target is to forecast the travel time for 70% of each flight bags (T70).6. T90 is similar to T70. It represents the travel time for 90% of each flight bags (T90). The level of uncertainty for T90 is higher than T70.

Six case studies

5-fold cross validation for

optimal NN structure

Each case study repeats

five times

Construction of NN-Based PIs for Forecast Uncertainty Modeling

Parameters for PSO and CWC

Median CWC vs. NN structure for DBT

Construction of NN-Based PIs for Forecast Uncertainty Modeling

CWC of the gbest particle in each generation of PSO. The constructed PIs of the 6 case studies.

0 200 400 600 8000

1

2#1-Ding10

Iterations

Gbe

st-C

WC

0 100 200 300 400 5000

1

2#2-HAS

IterationsG

best

-CW

C

0 100 200 3000

1

2#3-DBT

Iterations

Gbe

st-C

WC

0 100 200 3000

1

2#4-PBC

Iterations

Gbe

st-C

WC

0 100 200 300 400 5000

1

2#5-T70

Iterations

Gbe

st-C

WC

0 100 200 300 400 5000

1

2#6-T90

Iterations

Gbe

st-C

WC

0 20 40 60-1

-0.50

0.51

PIs of #1-Ding10

Samples

PIs

0 20 40 60-1

-0.50

0.51

PIs of #2-HAS

Samples

PIs

0 20 40 60-1

-0.50

0.51

PIs of #3-DBT

Samples

PIs

0 20 40 60-1

-0.50

0.51

PIs of #4-PBC

Samples

PIs

Test Data

PIs

0 20 40 60-1

-0.50

0.51

PIs of #5-T70

SamplesP

Is0 20 40 60

-1-0.5

00.5

1

PIs of #6-T90

Samples

PIs

The cost function can converge to a sufficient small CWC.

A sharp drop at the beginning, plateaus in the middle, finally (near) optimal

Indication: the strong searching ability of the PSO + mutation operation

For all case studies, the assigned confidence level (90%) can be satisfied. The median PINAWs are also smaller compared with benchmarks Percentage CWCs improvement: 26.55%, 28.91%, 29.04%, 18.93%, 41.12%

and 6.60% All PI construction time for test samples are less than 0.1ms, so the

algorithm is fast and efficient. In conclusion, the proposed PSO-based LUBE method can construct higher

quality PIs in a simpler and faster manner.

PI evaluation indices and construction time for test samples

Construction of NN-Based PIs for Forecast Uncertainty Modeling

Uncertainty Modeling in Smart Grids with Intermittent Renewable Generation

Datasets and Correlation Analysis

Real load data from Singapore (SG) & New South Wale (NSW), Australia;

Five years’ data, from Year 2007 to 2011, half an hour, each day 48 load points;

Correlation analysis: first three years, validation and test: fourth year and last year data;

Input selection: PACF (Partial Autocorrelation function) and ACF (Autocorrelation function), first seasonal differenced: y(t)-y(t-48*7), (T=48*7).

PIs for Electrical Load Forecasting

Datasets and Correlation Analysis

One week (48*7=336 points) ahead load forecasting, the top 15 peak lagged values as the inputs of the NN.

PIs for Electrical Load Forecasting

Original Load

Time

Load

0 10000 30000 50000

3000

4000

5000

6000

0 10000 30000 50000

-2000

01000

First seasonal differenced

Time

Load

0 500 1000 1500 2000-0

.20.2

0.6

1.0

Lag

Part

ial A

CF

Year 2007-2009

0 500 1000 1500 2000

-0.4

0.0

0.4

0.8

Lag

AC

F

Year 2007-2009

Correlation analysis of SG load

PACF results of SG and NSW load

H. Quan, D. Srinivasan, and A. Khosravi, “Uncertainty handling using neural network-based prediction intervals for

electrical load forecasting,” Energy, vol. 73, pp. 916-925, Aug. 2014.

Determination of Optimal Structure of the NN

Two hidden-layered NN, where 1≤n1≤10, 1≤n2≤10; Singapore load, the best NN structure is 16-5-1-2, NSW load, the

best NN structure is 16-8-1-2.

PIs for Electrical Load Forecasting

NN structures of SG and NSW load data

02 4 6

8 10

0246810

0.17

0.18

0.19

0.2

0.21

0.22

0.23

n1n2

PIN

AW

of

valid

ation d

ata

set

0.175

0.18

0.185

0.19

0.195

0.2

0.205

0.21

0.215

02

46

810

02

46

810

0.26

0.265

0.27

0.275

0.28

0.285

0.29

0.295

n1n2

PIN

AW

of

valid

ation d

ata

set

0.265

0.27

0.275

0.28

0.285

H. Quan, D. Srinivasan, and A. Khosravi, “Uncertainty handling using neural network-based prediction intervals for

electrical load forecasting,” Energy, vol. 73, pp. 916-925, Aug. 2014.

Constrained single objective optimization

PIs for Load Electrical Forecasting

0 50 100 150 200 250 300 3503500

4000

4500

5000

5500

6000

6500

Time

Upp

er-lo

wer

-tes

t bo

und

and

the

Tes

t D

ata

upper-bound-test

lower-bound-testTest Data

The PICPs for both SG and NSW load are high (>90%)

H. Quan, D. Srinivasan, and A. Khosravi, “Uncertainty handling using neural network-based prediction intervals for

electrical load forecasting,” Energy, vol. 73, pp. 916-925, Aug. 2014.

Variability of solar power output

There is a great deal of uncertainty associated with solar power generation

The variable nature of wind power

The output from a wind farm can be highly unpredictable

Uncertainty Modeling in Smart Grids with Intermittent Renewable Generation

Uncertainty Representation in Smart Grids

Solar generating sources

Solar irradiance distribution

Power outputs

Wind generating sources

Wind speed distribution

Uncertain power curve

Empirical power curve

Beta probability distributions (solar)

Weibull probability distributions (wind)

Uncertainty Modeling in Smart Grids with Intermittent Renewable Generation High uncertainty of wind and solar power has significant

impact on power system operation, economics, and reliability Existing forecasting methods cannot adequately

represent this uncertainty PIs for decision making and risk assessment

To develop advanced uncertainty modeling methods for forecasting;

To incorporate renewable generation forecast uncertainties into stochastic decision making and risk assessment.

Improved PSO-based LUBE method, PSO associated with mutation

operator

Different types of prediction tasks, including electrical load and wind

power generation forecasts are implemented

Outperforms ARIMA, exponential smoothing (ES) and naive models

Implementation is straightforward and much easier; PI construction

time is much shorter than traditional methods.

Uncertainty Handling Using NN-Based PIs for Wind Power Forecasting

Uncertainty Handling Using NN-Based PIs for Wind Power Forecasting

Primary problem: multi-objective, higher PICP and narrower width

Single-Objective Problem Formulation, cost function of CWC

Constrained Single-Objective Problem Formulation

Advantages: closer to the primary problem, fewer parameters

H. Quan, D. Srinivasan, and A. Khosravi, “Incorporating wind power forecast uncertainties into stochastic

unit commitment using neural network-based prediction intervals,” IEEE Transactions on Neural

Networks and Learning Systems, Nov. 2014.

Case Studies—Datasets and Correlation Analysis

Uncertainty Handling Using NN-Based PIs for Wind Power Forecasting

Dataset:

Capital Wind Farm, day-ahead forecasting

Correlation Analysis:

Seasonal Differencing

ACF and PACF analysis

NN input selection

The training process: simply converges

PICP of gbest: a few changes at beginning, it

quickly reaches the pre-assigned value.

PINRW of gbest: decreases sharply at the

beginning, then reduces step by step, finally its

optimal value.

Uncertainty Handling Using NN-Based PIs for Wind Power Forecasting

0 200 400 600 800 1000 1200 1400 1600 1800 200075

80

85

90

95

100

105

110

115

120

125

Iterations

PIC

P (

%)

and P

INR

W (

%)

of

gbest p

art

icle

s

PICP (%) of gbest

PINRW (%) of gbest

PICP and PINRW of gbest during training for Captl WF

Results

0 200 400 600 800 1000 1200 1400 1600 1800 200010

20

30

40

50

60

70

80

90

100

Iterations

PIC

P (

%)

and P

INR

W (

%)

of

gbest p

art

icle

s

PICP (%) of gbest

PINRW (%) of gbest

0 200 400 600 800 1000 1200 1400 1600 1800 200020

30

40

50

60

70

80

90

100

Iterations

PIC

P (

%)

and P

INR

W (

%)

of

gbest p

art

icle

s

PICP (%) of gbest

PINRW (%) of gbest

PICP and PINRW of gbest during training for NSW loadPICP and PINRW of gbest during training for SG load

Uncertainty Handling Using NN-Based PIs for Wind Power Forecasting

20 40 60 80 100 120 140 1600

20

40

60

80

100

120

140

Hours

Upper-

low

er-

test

bound a

nd t

he T

est

Data

upper-bound-test

lower-bound-testTest Data

Captl WF weekly generation and PIs for testing (1-7 Oct. 2010)

Constructed lower and upper bounds can cover the

real values in a great percentage.

The wind power uncertainty, much higher

34

Results and Discussions—Results Comparison

Uncertainty Handling Using NN-Based PIs for Load & Wind Power Forecasting

Test results of the Proposed Method and Benchmark Models CWC Percentage Improvements

Strong repeatability and stability

For all cases, the assigned confidence level 90%

can be satisfied

The widths of PIs reflects the

level of uncertainty in the data

PI construction time is very short

Significant improvements to

benchmarks

H. Quan, D. Srinivasan, and A. Khosravi, “Short-term load and wind power forecasting using neural network-based prediction

intervals,” IEEE Transactions on Neural Networks and Learning Systems, vol. 25, no. 2, pp. 303-315, Feb. 2014.

The primary multi-objective problem is successfully transformed into a

constrained single-objective problem.

Not only the high PICP and narrow PINAW are obtained, but also the PI

construction time remains short.

In conclusion, the proposed PSO-based LUBE method constructs higher

quality PIs for load and wind power forecasts in a short time.

Uncertainty Handling Using NN-Based PIs for Load & Wind Power Forecasting

Incorporating Forecast Uncertainties into Stochastic Decision Making process

The computational framework quantifies all grid uncertainties The integration framework is validated on the stochastic scheduling Generation costs, reserves of different scheduling strategies, risk profiles are

considered

H. Quan, D. Srinivasan, and A. Khosravi, “A computational framework for uncertainty integration in stochastic unit commitment with intermittent renewable energy resources,” Applied Energy , 2016.

Scenario Generation from the Wind Power PIs

Incorporating Wind Power Forecast Uncertainties into Stochastic Decision Making

A list of PIs for day-ahead wind forecasting The fitted ECDF curve

5 10 15 200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Hours

Win

d P

ower

[%

of

Cap

acity

]

95%90%

85%

80%

75%70%

65%

60%

55%50%

45%

40%35%

30%

25%

20%15%

10%

5%

pred.meas.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Win

d P

ower

(p.

u.)

ECDF

discrete quantiles

fitted cvure

Obtain the discrete points on ECDF

Fitting the ECDF curve

Wind power prediction intervals

Decompose PIs into quantiles

Stochastic Model for Uncertainty

Integration, Solar and Wind

Net load

New Power Balance Constraint

GA-Based Solution Method

5 deterministic and 4 stochastic cases

Incorporating Solar Power Forecast Uncertainties into Stochastic Decision Making

5 10 15 200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Hours

Sol

ar P

ower

[%

of

Cap

acity

]

95%90%

85%

80%75%

70%

65%60%

55%

50%

45%40%

35%

30%25%

20%

15%10%

5%

pred.meas.

2 4 6 8 10 12 14 16 18 20 22 240

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Sol

ar P

ower

(p.

u.)

Time Horizon (hr)

scenarios

prediction

A list of PIs for day-ahead solar power forecasting

The generated 50 solar scenarios for 24 hours

A Computational Framework for Uncertainty Integration with Renewable Generation

Load, wind and solar power uncertainties,

generator outages are considered

Five deterministic and four stochastic case

studies, different UC and reserve strategies

Superimposed effect of uncertainties in

uncertainty integration

The overall costs are less due to solar power

penetration

Stochastic VS. deterministic model: more

robust

Power systems run higher level of risk in peak

load hours

Real time ED reserve of Det. cases.

Real time ED reserve of Stoch. cases.

2 4 6 8 10 12 14 16 18 20 22 24

50

100

150

200

250

300

350

400

450

500

Hours

Rea

l Tim

e E

D A

vaila

ble

Res

erve

for

Unc

erta

inty

Int

egra

tion

(MW

)

D1. no windD2. point forecast

D3. perfect forecast

D4. 80% quantile

D5. 20% quantileED reserve requirement

2 4 6 8 10 12 14 16 18 20 22 2450

100

150

200

250

300

350

400

450

500

Hours

Rea

l Tim

e E

D A

vaila

ble

Res

erve

for

Unc

erta

inty

Int

egra

tion

(MW

)

S1. 10% reserve

S2. add. 5% of load

S3. add. 50% point forecastS4. add. point forecast-10% quantile

ED reserve requirement

H. Quan, D. Srinivasan, and A. Khosravi, “Integration of renewable generation uncertainties into stochastic unit commitment considering reserve and risk: a comparative study,” Energy Journal, 2016.

Conclusions Prediction Intervals are powerful for quantifying forecast

uncertainties

Advanced Uncertainty Handling Methods for Forecasting

PSO-based LUBE method shows significant improvements

on quality of PIs and computational speed.

Pis can be effectively used for Incorporation of Forecast

Uncertainties in Decision Making

A computational framework for uncertainty integration in

Smart Grid has been developed; incorporates deterministic

and stochastic scheduling, and reserve strategies considering

risk