&Q&L& _- l5! $1

4 ._

ELSEVIER International Journal of Forecasting 10 (1994) 161-164

Research on forecasting

The International Journal of Forecasting pro- vides critiques of papers published elsewhere. The editors try to select recent papers that are likely to be of significant interest to the readers. Our review of each paper is sent to the author for comment prior to publication. Almost all authors respond with suggestions and these typi- cally lead to improvements in the critiques.

If you know of interesting papers or if you have published such a paper, please send a copy to the Editors for possible inclusion in this section of the International Journal of Forecast- ing. To obtain copies of the papers reviewed in this section, contact the authors of the original papers.

Neil A. Gershenfeld and Andreas S. Weigend, 1994, ‘The future of time series’, in: A.S. Weigend and N.A. Gershenfeld, eds., Time Series Prediction : Forecasting the Future and Understanding the Past, (Addison-Wesley, Read- ing, MA), l-70.

Readers of this journal will be familiar with the M- and M2-Competitions that compared the accuracy of different forecasting methods. The Santa Fe (hereafter SF) competition was run during 1991. It was organised by Neil Gershen- feld (a physicist) and Andreas Weigend (a physi- cist/computer scientist) under the auspices of the SF Institute in New Mexico. The results were presented at a workshop in SF in May, 1992 and published in 1994 in a 641-page book. This review considers only the opening review chapter

by the organisers of the competition who are also the book’s editors.

The M- and M2-Competitions considered 1001 and 29 series, respectively, while the SF-Compe- tition only considers six series. The M- and M2- Competitions were primarily concerned with linear time-series forecasting methods, such as exponential smoothing and the use of ARIMA models. The SF-Competition is particularly concerned with non-linear methods such as the use of neural networks. The data for the M- and M2-Competitions were mainly short series (be- tween 50 and 100 observations) from the areas of economics, marketing and finance. In contrast the SF-Competition considered very long series, such as the 34 000 physiological observations on a patient with sleep apnea (though Neil Gershen- feld tells me that some physicists still complained that these series were too short!). There are three physical time series, one artificially gener- ated non-linear series, and the musical score of Bach’s final unfinished fugue! The only econ- omic-type series consists of ten segments of 3000 points of the exchange rate between the Swiss Franc and the US dollar measured every one or two minutes. The M- and MZCompetitions were solely concerned with forecasting, while the SF- Competition was also concerned with modelling and system characterization.

The MZCompetition was carried out under genuine post-sample conditions in that forecasts were made of future observations and then the participants waited for the actual observations to come in so that comparisons could be made. The SF-Competition (like the M-Competition) used historical data and the organisers kept back a hold-out sample for three of the series from

Elsevier Science B.V.

162 Research on forecasting I International Journal of Forecasting 10 (1994) 161-164

competition participants (prediction tasks were not set for all series).

Back in 1991 I considered taking part in the SF-Competition but when I received the data, I found that five of the data sets were very long non-linear-looking series from areas outside my previous expertise. It seemed to me that model- ling was perhaps more important than forecast- ing. The single economic series was described at that time only as ‘a financial series’ and no background information was provided to ensure that participants could not cheat and look up the actual results. This is a serious problem using historical data. Because of the lack of contextual information and the unfamiliarity of the data, I decided not to take part.

Having said this, I still find this review and the results very interesting. Because of their differ- ent background and perspective, the authors’ summary of known results and background ma- terial is fascinating for what it includes (some of which is new to me) and for what it leaves it out. 1 received several new insights from reading the chapter and I recommend it to everyone inter- ested in time-series forecasting especially if they are interested in non-linear methods.

The introduction (section 1) discusses a num- ber of general and historical issues, including the distinction between what the authors call under- standing (essentially looking for mathematical models), and learning (essentially algorithms which can emulate the structure in a time series). They also distinguish between the three objec- tives: (i) forecasting; (ii) modelling; (iii) system characterization. The latter attempts to deter- mine fundamental properties of the system such as spectral properties and measures of complexi- ty (dimension). Section 2 provides an overview of the Competition. It describes and plots the data giving more information than was available to participants. Section 3 reviews linear models and goes on to discuss how and why they break down. Non-linear time-series models, such as threshold autoregressive models, are discussed very briefly. Instead the authors concentrate (Section 4) on state-space models and neural networks, and also consider their use in forecast- ing (Section 5). Section 6 discusses system

characterization including such difficult topics as entropy and the Lyapunov exponent. Section 7 discusses the future. It points out that for theo- retical, as well as empirical, reasons, there “can- not be a universal forecasting algorithm” but predicts that “a robust theory of non-linear time series prediction and analysis will emerge that will join spectrum analysis and linear filters in any scientist’s working toolkit”. (The remainder of the book, which will not be reviewed here, consists of five chapters describing the five real data sets, nine chapters describing approaches to prediction for one or more data sets, seven chapters describing time-series analysis and characterization for one or more data sets, and five more general chapters on “Practice and Promise” including “Forecasting in Economics” by Clive Granger.)

It is impossible to provide a detailed survey of such a long weighty paper (let alone the whole book). I will therefore simply pick out some statements and comments which were of par- ticular interest to me (note especially point 8).

1. “Failure to use common sense was readily apparent in many of the entries in the competi- tion.”

2. Predictions “based solely on visually ex- amining and extrapolating the training data did much worse than the best techniques, but also much better than the worst.”

3. “There was a general failure of simplistic ‘black-box’ approaches - in all successful entries, exploratory data analysis preceded the algorithm application.”

4. “It does not make sense to attribute the linearity property to a time series, only to a system.”

5. “All the successful entries were fun- damentally non-linear. ”

6. Some non-linear results “go far beyond what is possible within the canon of linear systems analysis” but there are also “unpre- cedented opportunities for the analysis to go astray.” In particular “the’ best, as well as many of the worst, forecasts of Data Set A were obtained with neural networks.”

7. The neural network which gave best pre- dictions for Data Set A for “short” lead times

Research on forecasting I International Journal of Forecasting 10 (1994) 161-164 163

(meaning up to 100 steps ahead in this context!) did not work well at longer lead times. A state- space model gave much better forecasts for up to 500 steps ahead.

8. The prediction methods that work well for data sets A and D fail for the financial time series which is close to pure randomness. Here there is a “crucial difference between training set and test set performance” and “out-of-sample predictions are on average worse than chance.”

9. One successful set of predictions for Data Set D used 100 hours of computer time!

10. Autoregressive models can be thought of as having “an internal memory.”

11. Engineers use the description “finite im- pulse response (FIR) filter” for the “Moving Average Model.” The different terminology used in different disciplines for the same thing is potentially confusing.

- Chris Chatfield [Neil A. Gershenfeld, MIT Media Laboratory, 20 Ames Street, Cambridge, MA 02139, USA; Andreas S. Weigend, Computer Science Dept., University of Colorado (Box 430), Boulder, CO

80309-0439, USA]

SSDI 0169-2070(94)00526-I

Chung-ki Min and Arnold Zellner, 1993, “Bayesian and non-Bayesian methods for com- bining models and forecasts with applications to forecasting international growth rates”, Journal of Econometrics 56, 89-118. This article forms part of an on-going study that uses econometric models with a simple economic structure, but the parameters are estimated using a variety of schemes of varying complexity. The article is unusual in economics in that it pays considerable attention to forecasting accuracy and the econometric innovations find their main justification in an attempt to improve prediction. The authors forecast annual output growth rates for 18 countries year by year from 1974 to 1987 using data from 1951 to 1973 to start up the analysis. They utilize two basic models; (i) an autoregressive model of order three containing

lagging indicators of the past growth in country- specific real stock prices, the median growth across countries and the country-specific growth in real money, denoted by ARLZ; and (ii) the ARLZ model expanded to include a measure of current world income growth, denoted by ARLII WI. The world income variable WI is forecast using lags, real world stock prices and real world money growth. Two distinct assumptions are made concerning the parameters in these models: they are fixed (FP), or alternatively they vary according to a random walk (TVP). These two models of the parameters were estimated using conventional (Bayesian) techniques and by pooled estimators as follows. If pi, is the parame- ter vector for the ith country at time 1, then it can be modelled as follows:

where 0, =8,_, +nr and /3,! = (PI,, pi,, . . . , p;,). In effect the parameter vectors from the different countries are assumed to be random with a common mean, which will vary (or not) according to whether the parameters are though of as fixed or time varying.

The authors also add in a third-order auto- regressive benchmark model. The comparisons made are as follows: - FP versus TVP for the ARLI and ARLZi WI

models. - unpooled versus pooled estimation. - model selection using Bayesian posterior odds. -combined methods with the methods weighted

by the posterior odds ratio. - combined methods with regression weights

chosen using various constraining schemes (e.g. no intercept, weights summing to one). The forecasting literature is as yet unclear as

to the relative performance of the various ap- proaches. My own reading suggests that the evidence on fixed parameters versus time-varying parameters is moot, but that combining should lead to improvement.

The methods were carefully compared based on their individual country performance mea- sured by root mean squared errors for the years 1974-1987, and the distribution of these (par- ticularly the median).