Presentation title goes here · Example Data •In this framework forecast horizon becomes another...
Transcript of Presentation title goes here · Example Data •In this framework forecast horizon becomes another...
Session Goals
Motivation
Results Preview
Background
Tuning Models: Why and How
Primer on K-Fold Cross-Validation (K=4)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Fold 1
Fold 2
Fold 3
Fold 4Test DataValidation Fold 4
Validation Fold 3
Validation Fold 2Training Fold 2
Training Fold 3 Training Fold 3
Training Fold 4
Training Fold 2 Test Data
Test Data
Entire Dataset
Test DataValidation Fold 1Training Fold 1
Cross-Validation for Time SeriesApr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017
Apr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017
Apr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017
Not Used
Apr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017
Apr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017
Test DataNot Used
Entire Dataset
Fold 3Test Data
Fold 4Test Data
Training Fold 1
Training Fold 2
Training Fold 3
Training Fold 4
Validation Fold 1
Validation Fold 2
Validation Fold 3
Validation Fold 4
Not Used Test DataFold 1
Fold 2
Application Details
Using Out-of-Fold Forecasts as Features
Results• Mean absolute error:
• 𝑚𝑎𝑒 = Τσ𝑖=1𝐼 𝑦𝑖 − 𝑓𝑖 𝐼
• Displayed are improvements relative to mean forecast
• Mean is average of seasonal naïve, ARIMA, ETS, Elastic Net, and k-NN
• Ensemble: forecast from regression model with individual forecasts are features and revenue is target
• Numbers displayed are % differences between methods
• E.g. for Device 1, Country 1:𝑚𝑎𝑒𝐴𝑅𝐼𝑀𝐴
𝑚𝑎𝑒𝑀𝑒𝑎𝑛− 1 = −8.43%
Device Country S.Naive ARIMA ETS Elastic Net k-NN Ensemble
1 1 -2.28 -8.43 100.46 -16.89 -16.76 -82.45
2 -6.70 -6.52 100.47 -34.30 26.71 -70.30
3 -14.68 -32.72 -32.30 332.31 -18.18 -78.87
4 -27.11 -14.78 116.93 24.98 5.52 -74.18
5 -36.25 -25.23 -15.59 143.15 8.45 -74.84
6 -0.42 -8.48 7.92 104.55 -2.36 -29.49
7 -1.92 13.86 80.00 -15.92 69.85 -46.26
8 18.55 10.84 160.08 175.02 63.96 -50.27
Average -8.85 -8.93 64.75 89.11 17.15 -63.33
2 1 12.25 -0.94 132.48 -44.36 -32.14 -91.92
2 26.89 37.97 -20.90 57.00 41.67 -74.33
3 25.08 5.59 153.18 -47.53 -57.76 -80.30
4 38.15 203.74 227.58 3.70 -45.91 -71.69
5 32.80 178.00 90.10 140.05 -23.77 -53.79
6 36.17 100.83 170.08 -15.40 -37.35 -73.77
7 27.36 50.57 160.55 -82.08 -82.59 -89.11
Average 28.39 82.25 130.44 1.63 -33.98 -76.42
Takeaways
Example Data• In this framework forecast horizon
becomes another feature
• Other features include year, quarter, and month, perhaps trend
• Must include lags based on horizon to avoid peeking into the future
• Can use rolling forecasts from univariate time series models, e.g. ETS or ARIMA, as features
• “Driver” time series can be handled similarly
• e.g. Bing query data
Index 2016-M01 2016-M02 2016-M03 2016-M04 2016-M05 2016-M06
Revenue Y1 Y2 Y3 Y4 Y5 Y6
Month Horizon Revenue Lag 1 Lag 2 ETS Fcst
2016-M04 1 Y4 Y3 Y2 F_4|3
2016-M04 2 Y4 Y2 Y1 F_4|2
2016-M04 3 Y4 Y1 Y0 F_4|1
2016-M05 1 Y5 Y4 Y3 F_5|4
2016-M05 2 Y5 Y3 Y2 F_5|3
2016-M05 3 Y5 Y2 Y1 F_5|2
2016-M06 1 Y6 Y5 Y4 F_6|5
2016-M06 2 Y6 Y4 Y3 F_6|4
2016-M06 3 Y6 Y3 Y2 F_6|3
Cross-Validation with HorizonsApr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017
Apr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017
Horizon 1 Tr Tr Tr V1 V2 V3 V4 V5 V6 V7 V8 V9 V10
Horizon 2 Tr Tr Tr V1 V2 V3 V4 V5 V6 V7 V8 V9 V10
Horizon 3 Tr Tr Tr V1 V2 V3 V4 V5 V6 V7 V8 V9 V10
Tr Training V1 Validation, real quarter V2 Validation, pseudo-quarter
Entire Dataset