A Primer on Neural Networks for Forecasting
J. Stuart McMenamin · The Journal of Business Forecasting Methods & Systems · 1997
In most forecasting problems the optimal number of nodes appears to be between two and five ... in econometric model selection, some qualitative criteria such as signs of parameters and absolute magnitudes of elasticities may be used, whereas in neural networks, it is usually based on the bestfit... neural networks provide a flexible nonlinear modeling framework that can have significant advantages. Artificial neural network models are beginning to be used in the electric utility industry for short-term forecasting. The neural network framework provides a flexible function that can approximate a wide range of nonlinear processes. In forecasting problems where nonlinearities and variable interactions are important, neural networks can provide significant advantages. Despite these advantages, the topic of neural networks is surrounded by confusion and controversy. In part, this reflects the fact that a different language is used for neural networks than is used in the more familiar (to forecasters) area of econometrics. The main purpose of this article is to bridge this language gap. The discussion is put in Q & A format. Each question focuses on a specific issue or concept involved with specifying, estimating, or understanding neural networks. And the answers draw parallels, where possible, to elements of time series and econometric analysis. So, let's start with some basic questions. NEURAL NETWORK TERMINOLOGY Q. What exactly is an artificial neural network? A. Artificial neural network models are flexible nonlinear models. In the most general form, the type of neural network model typically used in forecasting can be written as follows: Y = Ft H,(X), H2(X), ..., HN(X) ] + u where Y is a dependent variable, X is a set of explanatory variables, F and the H's are the neural network functions, and u is the model error term. In the neural network language: The X's are called inputs Y is called the output The H functions are called the hidden layer activation functions F is called the output layer activation function Q. Is there a more specific form? A. Yes. In the specific form, that is normally used, F is linear in the H functions. The H functions are specified to be Sshaped curves using the function. In this case, the neural network model is described as follows: 1. Single output feed forward neural network 2. With one hidden layer and with multiple nodes in the hidden layer 3. With logistic activation functions in the hidden layer 4. With a linear activation function at the output layer Q. How does this relate to a linear regression model. A. The two main differences are that a linear regression model is linear in its parameters and there are no hidden layer functions in a regression model. The linear model takes the form: Y = XB + u In neural network terms, this is a single output feed forward system with no hidden layer and with a linear activation function at the output layer. In this sense, the linear regression model is a severely limited special case of the neural network framework. Q. Why is it called a feed forward neural network? A. The best way to answer this question is to draw the classic neural net diagram. As shown in Figure 1, the explanatory variables (X) enter at the bottom in the input layer. The logistic functions (H, and H2) appear in the hidden layer. And the result (Y) appears in the output layer. The idea is that the inputs feed into the functions in the hidden layer, and there is no feedback. Further, the functions in the hidden layer do not feed sideways into each other. Instead, they feed onward to the output layer. And there is no feedback, delayed, or otherwise, from the output layer to the hidden layer. The absence of feedback and the absence of interaction between hidden-layer functions makes it a feed forward system. …