Forecasting using

7. Transformations and adjustments

OTexts.com/fpp/2/4/

Forecasting using R 1

Rob J Hyndman

Outline

1 Exponential smoothing

2 Transformations

3 Adjustments

Forecasting using R Exponential smoothing 2

Exponential smoothing

ets() function

Automatically chooses a model by default usingthe AIC, AICc or BIC.

Can handle any combination of trend,seasonality and damping

Produces prediction intervals for every model

Ensures the parameters are admissible(equivalent to invertible)

Produces an object of class ets.

Forecasting using R Exponential smoothing 3

Exponential smoothing

ets() function

Automatically chooses a model by default usingthe AIC, AICc or BIC.

Can handle any combination of trend,seasonality and damping

Produces prediction intervals for every model

Ensures the parameters are admissible(equivalent to invertible)

Produces an object of class ets.

Forecasting using R Exponential smoothing 3

Exponential smoothing

ets() function

Automatically chooses a model by default usingthe AIC, AICc or BIC.

Can handle any combination of trend,seasonality and damping

Produces prediction intervals for every model

Ensures the parameters are admissible(equivalent to invertible)

Produces an object of class ets.

Forecasting using R Exponential smoothing 3

Exponential smoothing

ets() function

Automatically chooses a model by default usingthe AIC, AICc or BIC.

Can handle any combination of trend,seasonality and damping

Produces prediction intervals for every model

Ensures the parameters are admissible(equivalent to invertible)

Produces an object of class ets.

Forecasting using R Exponential smoothing 3

Exponential smoothing

ets() function

Automatically chooses a model by default usingthe AIC, AICc or BIC.

Can handle any combination of trend,seasonality and damping

Produces prediction intervals for every model

Ensures the parameters are admissible(equivalent to invertible)

Produces an object of class ets.

Forecasting using R Exponential smoothing 3

Exponential smoothing

ets objects

Methods: coef(), plot(), summary(),

residuals(), fitted(), simulate()

and forecast()

plot() function shows time plots of the

original time series along with the

extracted components (level, growth

and seasonal).

Forecasting using R Exponential smoothing 4

Exponential smoothing

ets objects

Methods: coef(), plot(), summary(),

residuals(), fitted(), simulate()

and forecast()

plot() function shows time plots of the

original time series along with the

extracted components (level, growth

and seasonal).

Forecasting using R Exponential smoothing 4

Exponential smoothing

Forecasting using R Exponential smoothing 5

200

400

600

obse

rved

250

350

450

leve

l

0.99

01.

005

slop

e

0.9

1.1

1960 1970 1980 1990 2000 2010

seas

on

Time

Decomposition by ETS(M,Md,M) methodplot(fit)

Goodness-of-fit

> accuracy(fit)ME RMSE MAE MPE MAPE MASE

0.17847 15.48781 11.77800 0.07204 2.81921 0.20705

> accuracy(fit2)ME RMSE MAE MPE MAPE MASE

-0.11711 15.90526 12.18930 -0.03765 2.91255 0.21428

Forecasting using R Exponential smoothing 6

Forecast intervals

Forecasting using R Exponential smoothing 7

Forecasts from ETS(M,Md,M)

1995 2000 2005 2010

300

350

400

450

500

550

600

> plot(forecast(fit,level=c(50,80,95)))

Forecast intervals

Forecasting using R Exponential smoothing 7

Forecasts from ETS(M,Md,M)

1995 2000 2005 2010

300

350

400

450

500

550

600

> plot(forecast(fit,fan=TRUE))

Exponential smoothing

ets() function also allows refitting model to newdata set.

> usfit <- ets(usnetelec[1:45])> test <- ets(usnetelec[46:55], model = usfit)

> accuracy(test)ME RMSE MAE MPE MAPE MASE

-3.35419 58.02763 43.85545 -0.07624 1.18483 0.52452

> accuracy(forecast(usfit,10), usnetelec[46:55])ME RMSE MAE MPE MAPE MASE

40.7034 61.2075 46.3246 1.0980 1.2620 0.6776

Forecasting using R Exponential smoothing 8

Unstable models

ETS(M,M,A)

ETS(M,Md,A)

ETS(A,N,M)

ETS(A,A,M)

ETS(A,Ad,M)

ETS(A,M,N)

ETS(A,M,A)

ETS(A,M,M)

ETS(A,Md,N)

ETS(A,Md,A)

ETS(A,Md,M)Forecasting using R Exponential smoothing 9

Unstable models

ETS(M,M,A)

ETS(M,Md,A)

ETS(A,N,M)

ETS(A,A,M)

ETS(A,Ad,M)

ETS(A,M,N)

ETS(A,M,A)

ETS(A,M,M)

ETS(A,Md,N)

ETS(A,Md,A)

ETS(A,Md,M)Forecasting using R Exponential smoothing 9

In practice, the modelswork fine for short- tomedium-term forecastsprovided the data arestrictly positive.

Forecastability conditions

ets(y, model="ZZZ", damped=NULL, alpha=NULL,beta=NULL, gamma=NULL, phi=NULL,additive.only=FALSE,lower=c(rep(0.0001,3),0.80),upper=c(rep(0.9999,3),0.98),opt.crit=c("lik","amse","mse","sigma"),nmse=3,bounds=c("both","usual","admissible"),ic=c("aic","aicc","bic"), restrict=TRUE)

Forecasting using R Exponential smoothing 10

The magic forecast() function

forecast returns forecasts when applied to anets object (or the output from many other timeseries models).

If you use forecast directly on data, it willselect an ETS model automatically and thenreturn forecasts.

Forecasting using R Exponential smoothing 11

The magic forecast() function

forecast returns forecasts when applied to anets object (or the output from many other timeseries models).

If you use forecast directly on data, it willselect an ETS model automatically and thenreturn forecasts.

Forecasting using R Exponential smoothing 11

Outline

1 Exponential smoothing

2 Transformations

3 Adjustments

Forecasting using R Transformations 12

Transformations to stabilize the variance

If the data show different variation at different levels of theseries, then a transformation can be useful.Denote original observations as y1, . . . , yn and transformedobservations as w1, . . . ,wn.

Mathematical transformations for stabilizing variation

Square root wt =√yt ↓

Cube root wt = 3√yt Increasing

Logarithm wt = log(yt) strength

Logarithms, in particular, are useful because they are more

interpretable: changes in a log value are relative (percent)

changes on the original scale.

Forecasting using R Transformations 13

Transformations to stabilize the variance

If the data show different variation at different levels of theseries, then a transformation can be useful.Denote original observations as y1, . . . , yn and transformedobservations as w1, . . . ,wn.

Mathematical transformations for stabilizing variation

Square root wt =√yt ↓

Cube root wt = 3√yt Increasing

Logarithm wt = log(yt) strength

Logarithms, in particular, are useful because they are more

interpretable: changes in a log value are relative (percent)

changes on the original scale.

Forecasting using R Transformations 13

Transformations to stabilize the variance

If the data show different variation at different levels of theseries, then a transformation can be useful.Denote original observations as y1, . . . , yn and transformedobservations as w1, . . . ,wn.

Mathematical transformations for stabilizing variation

Square root wt =√yt ↓

Cube root wt = 3√yt Increasing

Logarithm wt = log(yt) strength

Logarithms, in particular, are useful because they are more

interpretable: changes in a log value are relative (percent)

changes on the original scale.

Forecasting using R Transformations 13

Transformations to stabilize the variance

If the data show different variation at different levels of theseries, then a transformation can be useful.Denote original observations as y1, . . . , yn and transformedobservations as w1, . . . ,wn.

Mathematical transformations for stabilizing variation

Square root wt =√yt ↓

Cube root wt = 3√yt Increasing

Logarithm wt = log(yt) strength

Logarithms, in particular, are useful because they are more

interpretable: changes in a log value are relative (percent)

changes on the original scale.

Forecasting using R Transformations 13

Transformations

Forecasting using R Transformations 14

Square root electricity production

Year

1960 1970 1980 1990

4060

8010

012

0

Transformations

Forecasting using R Transformations 14

Cube root electricity production

Year

1960 1970 1980 1990

1214

1618

2022

24

Transformations

Forecasting using R Transformations 14

Log electricity production

Year

1960 1970 1980 1990

7.5

8.0

8.5

9.0

9.5

Transformations

Forecasting using R Transformations 14

Inverse electricity production

Year

1960 1970 1980 1990

−8e

−04

−6e

−04

−4e

−04

−2e

−04

Box-Cox transformations

Each of these transformations is close to a memberof the family of Box-Cox transformations:

wt =

{log(yt), λ = 0;(yλt − 1)/λ, λ 6= 0.

λ = 1: (No substantive transformation)

λ = 12: (Square root plus linear transformation)

λ = 0: (Natural logarithm)

λ = −1: (Inverse plus 1)

Forecasting using R Transformations 15

Box-Cox transformations

Each of these transformations is close to a memberof the family of Box-Cox transformations:

wt =

{log(yt), λ = 0;(yλt − 1)/λ, λ 6= 0.

λ = 1: (No substantive transformation)

λ = 12: (Square root plus linear transformation)

λ = 0: (Natural logarithm)

λ = −1: (Inverse plus 1)

Forecasting using R Transformations 15

Box-Cox transformations

Each of these transformations is close to a memberof the family of Box-Cox transformations:

wt =

{log(yt), λ = 0;(yλt − 1)/λ, λ 6= 0.

λ = 1: (No substantive transformation)

λ = 12: (Square root plus linear transformation)

λ = 0: (Natural logarithm)

λ = −1: (Inverse plus 1)

Forecasting using R Transformations 15

Box-Cox transformations

Each of these transformations is close to a memberof the family of Box-Cox transformations:

wt =

{log(yt), λ = 0;(yλt − 1)/λ, λ 6= 0.

λ = 1: (No substantive transformation)

λ = 12: (Square root plus linear transformation)

λ = 0: (Natural logarithm)

λ = −1: (Inverse plus 1)

Forecasting using R Transformations 15

Box-Cox transformations

Each of these transformations is close to a memberof the family of Box-Cox transformations:

wt =

{log(yt), λ = 0;(yλt − 1)/λ, λ 6= 0.

λ = 1: (No substantive transformation)

λ = 12: (Square root plus linear transformation)

λ = 0: (Natural logarithm)

λ = −1: (Inverse plus 1)

Forecasting using R Transformations 15

Box-Cox transformations

Forecasting using R Transformations 16

Box-Cox transformations

yλt for λ close to zero behaves like logs.If some yt = 0, then must have λ > 0if some yt < 0, no power transformation ispossible unless all yt adjusted by adding aconstant to all values.Choose a simple value of λ. It makesexplanation easier.Results are relatively insensitive to value of λOften no transformation (λ = 1) needed.Transformation often makes little difference toforecasts but has large effect on PI.Choosing λ = 0 is a simple way to forceforecasts to be positive

Forecasting using R Transformations 17

Box-Cox transformations

yλt for λ close to zero behaves like logs.If some yt = 0, then must have λ > 0if some yt < 0, no power transformation ispossible unless all yt adjusted by adding aconstant to all values.Choose a simple value of λ. It makesexplanation easier.Results are relatively insensitive to value of λOften no transformation (λ = 1) needed.Transformation often makes little difference toforecasts but has large effect on PI.Choosing λ = 0 is a simple way to forceforecasts to be positive

Forecasting using R Transformations 17

Box-Cox transformations

yλt for λ close to zero behaves like logs.If some yt = 0, then must have λ > 0if some yt < 0, no power transformation ispossible unless all yt adjusted by adding aconstant to all values.Choose a simple value of λ. It makesexplanation easier.Results are relatively insensitive to value of λOften no transformation (λ = 1) needed.Transformation often makes little difference toforecasts but has large effect on PI.Choosing λ = 0 is a simple way to forceforecasts to be positive

Forecasting using R Transformations 17

Box-Cox transformations

yλt for λ close to zero behaves like logs.If some yt = 0, then must have λ > 0if some yt < 0, no power transformation ispossible unless all yt adjusted by adding aconstant to all values.Choose a simple value of λ. It makesexplanation easier.Results are relatively insensitive to value of λOften no transformation (λ = 1) needed.Transformation often makes little difference toforecasts but has large effect on PI.Choosing λ = 0 is a simple way to forceforecasts to be positive

Forecasting using R Transformations 17

Box-Cox transformations

yλt for λ close to zero behaves like logs.If some yt = 0, then must have λ > 0if some yt < 0, no power transformation ispossible unless all yt adjusted by adding aconstant to all values.Choose a simple value of λ. It makesexplanation easier.Results are relatively insensitive to value of λOften no transformation (λ = 1) needed.Transformation often makes little difference toforecasts but has large effect on PI.Choosing λ = 0 is a simple way to forceforecasts to be positive

Forecasting using R Transformations 17

Box-Cox transformations

yλt for λ close to zero behaves like logs.If some yt = 0, then must have λ > 0if some yt < 0, no power transformation ispossible unless all yt adjusted by adding aconstant to all values.Choose a simple value of λ. It makesexplanation easier.Results are relatively insensitive to value of λOften no transformation (λ = 1) needed.Transformation often makes little difference toforecasts but has large effect on PI.Choosing λ = 0 is a simple way to forceforecasts to be positive

Forecasting using R Transformations 17

Box-Cox transformations

yλt for λ close to zero behaves like logs.If some yt = 0, then must have λ > 0if some yt < 0, no power transformation ispossible unless all yt adjusted by adding aconstant to all values.Choose a simple value of λ. It makesexplanation easier.Results are relatively insensitive to value of λOften no transformation (λ = 1) needed.Transformation often makes little difference toforecasts but has large effect on PI.Choosing λ = 0 is a simple way to forceforecasts to be positive

Forecasting using R Transformations 17

Box-Cox transformations

yλt for λ close to zero behaves like logs.If some yt = 0, then must have λ > 0if some yt < 0, no power transformation ispossible unless all yt adjusted by adding aconstant to all values.Choose a simple value of λ. It makesexplanation easier.Results are relatively insensitive to value of λOften no transformation (λ = 1) needed.Transformation often makes little difference toforecasts but has large effect on PI.Choosing λ = 0 is a simple way to forceforecasts to be positive

Forecasting using R Transformations 17

Back-transformation

We must reverse the transformation (orback-transform) to obtain forecasts on the originalscale. The reverse Box-Cox transformations aregiven by

yt =

{exp(wt), λ = 0;(λWt + 1)1/λ, λ 6= 0.

plot(BoxCox(elec,lambda=1/3))fit <- snaive(elec, lambda=1/3)plot(fit)plot(fit, include=120)

Forecasting using R Transformations 18

Back-transformation

We must reverse the transformation (orback-transform) to obtain forecasts on the originalscale. The reverse Box-Cox transformations aregiven by

yt =

{exp(wt), λ = 0;(λWt + 1)1/λ, λ 6= 0.

plot(BoxCox(elec,lambda=1/3))fit <- snaive(elec, lambda=1/3)plot(fit)plot(fit, include=120)

Forecasting using R Transformations 18

Automated Box-Cox transformations

BoxCox.lambda(elec)

This attempts to balance the seasonalfluctuations and random variation across theseries.

Always check the results.

A low value of λ can give extremely largeprediction intervals.

Forecasting using R Transformations 19

Automated Box-Cox transformations

BoxCox.lambda(elec)

This attempts to balance the seasonalfluctuations and random variation across theseries.

Always check the results.

A low value of λ can give extremely largeprediction intervals.

Forecasting using R Transformations 19

Automated Box-Cox transformations

BoxCox.lambda(elec)

This attempts to balance the seasonalfluctuations and random variation across theseries.

Always check the results.

A low value of λ can give extremely largeprediction intervals.

Forecasting using R Transformations 19

Automated Box-Cox transformations

BoxCox.lambda(elec)

This attempts to balance the seasonalfluctuations and random variation across theseries.

Always check the results.

A low value of λ can give extremely largeprediction intervals.

Forecasting using R Transformations 19

ETS and transformations

A Box-Cox transformation followed by anadditive ETS model is often better than an ETSmodel without transformation.

A Box-Cox transformation followed by STL +ETS is often better than an ETS model withouttransformation.

It makes no sense to use a Box-Coxtransformation and a non-additive ETS model.

Forecasting using R Transformations 20

ETS and transformations

A Box-Cox transformation followed by anadditive ETS model is often better than an ETSmodel without transformation.

A Box-Cox transformation followed by STL +ETS is often better than an ETS model withouttransformation.

It makes no sense to use a Box-Coxtransformation and a non-additive ETS model.

Forecasting using R Transformations 20

ETS and transformations

A Box-Cox transformation followed by anadditive ETS model is often better than an ETSmodel without transformation.

A Box-Cox transformation followed by STL +ETS is often better than an ETS model withouttransformation.

It makes no sense to use a Box-Coxtransformation and a non-additive ETS model.

Forecasting using R Transformations 20

Outline

1 Exponential smoothing

2 Transformations

3 Adjustments

Forecasting using R Adjustments 21

Calendar adjustments

Some of the variation in a time series may be due tothe variation in the number of trading days eachmonth. It is a good idea to adjust for this knownsource of variation to allow study of otherinteresting features.

Month length

Trading day

Forecasting using R Adjustments 22

Calendar adjustments

Some of the variation in a time series may be due tothe variation in the number of trading days eachmonth. It is a good idea to adjust for this knownsource of variation to allow study of otherinteresting features.

Month length

Trading day

Forecasting using R Adjustments 22

Month length adjustment

If this is not removed, it shows up as a seasonaleffect, which may not cause problems though itdoes make any seasonal pattern hard to interpret. Itis easily adjusted for:

y∗t = yt ×no. of days in an average month

no. of days in month t

= yt ×365.25/12

no. of days in month t

where yt has already been transformed if necessary.

monthdays gives the number of days in each monthor quarter.

Forecasting using R Adjustments 23

Month length adjustment

If this is not removed, it shows up as a seasonaleffect, which may not cause problems though itdoes make any seasonal pattern hard to interpret. Itis easily adjusted for:

y∗t = yt ×no. of days in an average month

no. of days in month t

= yt ×365.25/12

no. of days in month t

where yt has already been transformed if necessary.

monthdays gives the number of days in each monthor quarter.

Forecasting using R Adjustments 23

Trading day adjustmentoccurs in monthly data when there is also aweekly cycle, since proportions of various daysin given month vary from year to year.number of trading days is predictable, buteffects of various days are unknown.Simplest case: All trading days assumed tohave same effect.

y∗t = yt ×no. of trading days in average month

no. of trading days in month t.

where yt has already been adjusted for monthlength and transformed if necessary.If weekly cycle more complex, these effectsmust be estimated from data.

Forecasting using R Adjustments 24

Trading day adjustmentoccurs in monthly data when there is also aweekly cycle, since proportions of various daysin given month vary from year to year.number of trading days is predictable, buteffects of various days are unknown.Simplest case: All trading days assumed tohave same effect.

y∗t = yt ×no. of trading days in average month

no. of trading days in month t.

where yt has already been adjusted for monthlength and transformed if necessary.If weekly cycle more complex, these effectsmust be estimated from data.

Forecasting using R Adjustments 24

Trading day adjustmentoccurs in monthly data when there is also aweekly cycle, since proportions of various daysin given month vary from year to year.number of trading days is predictable, buteffects of various days are unknown.Simplest case: All trading days assumed tohave same effect.

y∗t = yt ×no. of trading days in average month

no. of trading days in month t.

where yt has already been adjusted for monthlength and transformed if necessary.If weekly cycle more complex, these effectsmust be estimated from data.

Forecasting using R Adjustments 24

Trading day adjustmentoccurs in monthly data when there is also aweekly cycle, since proportions of various daysin given month vary from year to year.number of trading days is predictable, buteffects of various days are unknown.Simplest case: All trading days assumed tohave same effect.

y∗t = yt ×no. of trading days in average month

no. of trading days in month t.

where yt has already been adjusted for monthlength and transformed if necessary.If weekly cycle more complex, these effectsmust be estimated from data.

Forecasting using R Adjustments 24

Explainable variation

Examples:

Calendar variation

Increasing population

Inflation

Strikes

Changes in government

Changes in law

Try and understand all possible sources of variationbefore modelling the time series.

Forecasting using R Adjustments 25

Explainable variation

Examples:

Calendar variation

Increasing population

Inflation

Strikes

Changes in government

Changes in law

Try and understand all possible sources of variationbefore modelling the time series.

Forecasting using R Adjustments 25

Explainable variation

Examples:

Calendar variation

Increasing population

Inflation

Strikes

Changes in government

Changes in law

Try and understand all possible sources of variationbefore modelling the time series.

Forecasting using R Adjustments 25

Explainable variation

Examples:

Calendar variation

Increasing population

Inflation

Strikes

Changes in government

Changes in law

Try and understand all possible sources of variationbefore modelling the time series.

Forecasting using R Adjustments 25

Explainable variation

Examples:

Calendar variation

Increasing population

Inflation

Strikes

Changes in government

Changes in law

Try and understand all possible sources of variationbefore modelling the time series.

Forecasting using R Adjustments 25

Explainable variation

Examples:

Calendar variation

Increasing population

Inflation

Strikes

Changes in government

Changes in law

Try and understand all possible sources of variationbefore modelling the time series.

Forecasting using R Adjustments 25

Explainable variation

Examples:

Calendar variation

Increasing population

Inflation

Strikes

Changes in government

Changes in law

Try and understand all possible sources of variationbefore modelling the time series.

Forecasting using R Adjustments 25

Explainable variation

Examples:

Calendar variation

Increasing population

Inflation

Strikes

Changes in government

Changes in law

Try and understand all possible sources of variationbefore modelling the time series.

Forecasting using R Adjustments 25