Nonlinear Dimensionality Reduction of Data by Deep Distributed Random Samplings
Xiao-Lei Zhang · arXiv (Cornell University) · 2014
Abstract—It is believed that if machine can learn human-level invariant semantic concepts from highly-variant real-world data, the real artificial intelligence is able to be touched. If we encode the data and concepts compactly by for example 0 and 1, the golden pathway is a problem of reducing high-dimensional codes to low-dimensional representations. Dimensionality reduction [1]–[10], as a fundamental problem of machine learning—one of the central fields of artificial intelligence, has been intensively studied, where classification and clustering are two special cases of dimensionality reduction that reduce high-dimensional data to discrete points. Multilayer neural networks [10]–[12] have demonstrated their great power on dimensionality reduction [10], which triggered the recent breakthrough of artificial intelligence. However, current multilayer neural networks are limited to large-scale problems, and their training methods [10], [12], [13] are so far complicated and time-consuming. Here we describe a simple multilayer network for dimensionality reduction that each layer of the network is a group of mutually independent k-centers clusterings. We find that the network can be trained successfully layer-by-layer by simply assigning the centers of each clustering by randomly sampled data points from the input. Our results show that the described simple method outperformed 7 well-known dimensionality reduction methods [3]–[5], [9], [10], [14] on both very small-scale biomedical data [15], [16] and large-scale image and document data [17]–[19], with much less training time than multilayer neural networks on large-scale data. Our findings imply that if properly designed, a very simple training method can make multilayer networks work well on a wide range of data. Furthermore, given the broad use of simple methods, the described method, which may be easily understood without the domain knowledge, will have a bright future in many branches of science. Index Terms—Bootstrap, deep learning, dimensionality reduc-tion, evolutionary computing, kernel methods, sparse coding. I.