Topological Properties of the Set of Functions Generated by Neural Networks of Fixed Size
Philipp Petersen, Mones Raslan, Felix Voigtlaender · Foundations of Computational Mathematics · 2020
Abstract We analyze the topological properties of the set of functions that can be implemented by neural networks of a fixed size. Surprisingly, this set has many undesirable properties. It is highly non-convex, except possibly for a few exotic activation functions. Moreover, the set is not closed with respect to $$L^p$$ Lp -norms, $$0 0$$ ε>0 , it is, regardless of the size of $$\varepsilon $$ ε , usually not possible to find weights $$w_1, w_2$$ w1,w2 close together such that each $$f_i$$ fi is realized by a neural network with weights $$w_i$$ wi . Overall, our findings identify potential causes for issues in the training procedure of deep learning such as no guaranteed convergence, explosion of parameters, and slow convergence.