Manifold Curvature Estimation for Neural Networks
Ali Sekmen, Bahadir Bilgin · 2022 IEEE International Conference on Big Data (Big Data) · 2022
This paper introduces a novel method for creating a metric to measure curvature of a discretized manifold. For each data point xion a manifold, a subspace ${{\mathcal{S}}_i}$ is matched using a number of neighboring points of xi. A local subspace is also matched to each neighboring point of xi. Then, a set of weighted angles between ${{\mathcal{S}}_i}$ and each neighboring local subspace are computed and the minimum of those weighted angles is used as a measure of curvature of the manifold at xi. The average curvature for all data points on the manifold is used as metric for the manifold’s curvature estimation. This research also uses the proposed metric to show that each layer of a neural network maps an input manifold to a flatter manifold during the training process. It is observed that each successive block in a neural network generates a manifold with less curvature than that of the previous layer. Another observation is that convolutional layers always generate flatter manifolds. Furthermore, it is shown that this metric can be used as a robustness measure for a neural network. The method has been tested successfully using two datasets MNIST and Extended YaleB datasets.