Efficient Approximation with Neural Networks: A Comparison of Gate Functions
Bhaskar DasGupta, Georg Schnitger · 2005
We compare di#erent gate functions in terms of the approximation power of their circuits. Evaluation criteria are circuit size s, circuit depth d and the approximation error e(s, d). Informally, gate functions # 1 and # 2 are called equivalent if {# 1 }-circuits of size s and depth d can be approximated by {# 2 }-circuits (and vice versa) of size poly(s), depth O(d) with approximation error e(s, d) = 2 -s . Our goal is to determine those gate functions that are equivalent to splines relative to this error model. The class of equivalent gate functions contains, among others, the exponential function, the natural logarithm, (non-polynomial) rational functions and (non-polynomial) roots. Newman's result, i.e., approximating | x | by rational functions, is obtained as a corollary of this equivalence result. Provably not equivalent are polynomials, the sine-function and linear splines. 1 Introduction We consider e#cient approximations of a given multivariate function f : [-1...