Why Neural Networks Are Computationally Efficient Approximators: An Explanation
Jaime Nava, Владик Крейнович · scholarworks - UTEP (The University of Texas at El Paso) · 2011
Many real-life dependencies can be reasonably accurately described by linear functions. If we want a more accurate description, we need to take non-linear terms into account. To take nonlinear terms into account, we can either explicitly add quadratic terms to the regression equation, or, alternatively, we can use a neural network with a non-linear activation function. At first glance, regression algorithms would work faster, but in practice, often, a neural network approximation turns out to be a more computationally efficient one. In this paper, we provide a reasonable explanation for this empirical fact. 1 Formulation of the Problem Practical need to find dependencies. In practice, it often occurs that we know (or conjecture) that a quantity y depends on quantities x1,..., xn, but we do not know the exact form of this dependence. In such situations, we must experimentally determine this dependence y = f(x1,..., xn). For that, in several (S) situations s = 1,..., N, we measure the values of both the dependent ( variable y and of the independent variables xi. Then, we use the results x (s) 1,..., x(s) n, y (s) of these measurements to find a function f(x1,..., xn) which is consistent with all these measurement results, i.e., for which y (s) ( ≈ f,..., x(s) x (s) 1 for all s from 1 to S. (The equality is usually approximate since the measurements are approximate and the value y is often only approximately determined by the values of the variables x1,..., xn.) n 1 First approximation: linear dependence. In many practical situations, the dependence f(x1,..., xn) is smooth: informally, this means that small changes in xi lead to equally small changes in y. In the first approximation, a smooth function can be approximated by its tangent, i.e., by a linear expression f(x1,..., xn) = c + n∑ i=1 ci · xi for appropriate coefficients c and ci. The task of ( estimating the values of these coefficients based on the measurement results,..., x(s) , y (s), i.e., based on the system of equations