Discovering and Visualizing Hierarchy in the Data
Kun Yang, Wing Hung Wong · arXiv (Cornell University) · 2014
Abstract—How to extract useful insights from data in a human perceivable manner is always a challenge when the dimension and amount of the data is large. Often, the data can be organized according to certain hierarchical structure that are stemmed either from data collection process or from the information and phenomena carried by the data itself. The current study attempts to discover and visualize these underlying hierarchies. Regarding each observation as a draw from a (hypothetical) multidimensional joint density, our first goal is to approximate this unknown density with a piecewise constant function over the binary partitioned sample space; our non-parametric approach makes no assumptions on the form of the density, such as assuming that it is Multivariate Gaussian, or that it is a mixture of a small number of Gaussians. Given the piecewise constant density function and its corresponding partitions of the sample space, our second goal is to construct a connected graph and build up a tree representation of the data from sub-level sets. To demonstrate that our method is a general data mining and visualization tool which can provide “multi-resolution ” summaries and reveal different levels of information of the data, we apply it to two real data sets from different fields.