The Universal Approximation Property: Characterizations, Existence, and a Canonical Topology for Deep-Learning
Anastasis Kratsios · arXiv (Cornell University) · 2019
The universal approximation property (UAP) of feed-forward neural networks is systematically studied for arbitrary families of functions in general function spaces. Two characterizations of the UAP are found, conditions for the existence of a small family of functions with the UAP are given, and a canonical topology guaranteeing that a set of functions has the UAP is explicitly constructed. These general results are applied to two concrete problems in learning theory. First, it is shown that neural network architectures with a sigmoid activation function achieving the values 0 and 1 are capable of approximating any set function between two Euclidean spaces for the canonical topology. As a second application of our results, it is shown that any continuous function accepting an arbitrary number of inputs can be approximated by a neural network receiving an arbitrary number of inputs. This makes these networks suitable for learning problems where the dimension of the data is diverging, such as in ultra-high dimensional situations.