The Power of Approximating: a Comparison of Activation Functions
Bhaskar DasGupta, Georg Schnitger · 1992
We compare activation functions in terms of the approximation power of their feedforward nets. We consider the case of analog as well as boolean input. 1 Introduction We consider efficient approximations of a given multivariate function f : [\\Gamma1; 1] m ! R by feedforward neural networks. We first introduce the notion of a feedforward net. Let \\Gamma be a class of real-valued functions, where each function is defined on some subset of R. A \\Gamma-net C is an unbounded fan-in circuit whose edges and vertices are labeled by real numbers. The real number assigned to an edge (resp. vertex) is called its weight (resp. its threshold). Moreover, to each vertex v an activation function fl v 2 \\Gamma is assigned. Finally, we assume that C has a single sink w. The net C computes a function f C : [\\Gamma1; 1] m ! R as follows. The components of the input vector x = (x 1 ; : : : ; xm ) 2 [\\Gamma1; 1] m are assigned to the sources of C. Let v 1 ; : : : ; vn be the immediate predecessors ...