Conical dimension as an intrisic dimension estimator and its applications
Xin Yang, Sébastien Michéa, Hongyuan Zha · 2007
Estimating the intrinsic dimension of a high-dimensional data set is a very challenging problem in manifold learning and several other application areas in data mining. In this paper we introduce a novel local intrinsic dimension estimator, conical dimension, for estimating the intrinsic dimension of a data set consisting of points lying in the proximity of a manifold. Under minimal sampling assumptions, we show that the conical dimension of sample points in a manifold is equal to the dimension of the manifold. The conical dimension enjoys several desirable properties such as linear conformal invariance and it can also handle manifolds with self-intersections as well as detect the boundary of manifolds. We develop algorithms for computing the conical dimension paying special attention to the numerical robustness issues. We apply the proposed algorithms to both synthetic and real-world data illustrating their robustness on noisy data sets with large curvatures.