Characterizing the Universal Approximation Property
Anastasis Kratsios · arXiv (Cornell University) · 2019
To better understand the approximation capabilities of various currently available neural network architectures, this paper studies the universal approximation property itself across a broad scope of function spaces. We characterize universal approximators, on most function space of practical interest, as implicitly decomposing that space into topologically regular subspaces on which a transitive dynamical system describes the architecture's structure. We obtain a simple criterion for constructing universal approximators as transformations of the feed-forward architecture and we show that every architecture, on most function spaces of practical interest, is approximately constructed in this way. Moreover, we show that most function spaces admit universal approximators built using a single function. The results are used to show that certain activation functions such as Leaky-ReLU, but not ReLU, create expressibility through depth by eventually mixing any two functions' open neighbourhoods. For those activation functions, we obtain improved approximation rates described in terms of the network breadth and depth. We show that feed-forward networks built using such activation functions can encode constraints into their final layers while simultaneously maintaining their universal approximation capabilities. We construct a modification of the feed-forward architecture, which can approximate any continuous function, with a controlled growth rate, uniformly on the entire domain space, and we show that the feed-forward architecture typically cannot.