Practical Macrostate Data Clustering
Brian S. White, David I. Shalloway · arXiv (Cornell University) · 2007
Spectral clustering methods have been shown to outperform traditional distance-based approaches, such as k-means and hierarchical clustering, based on their use of global information encoded in eigenvectors of a matrix describing inter-item relations. Macrostate data clustering [Korenblum and Shalloway, Phys. Rev. E, Volume 67, 2003] used an analogy to the dynamic coarse-graining of a stochastic system to construct a linear combination of eigenvectors that probabilistically assigned items to clusters. A ``minimum uncertainty criterion'' lead to an objective function that minimized the inherent fuzziness of the cluster assignments. The resulting non-linear optimization problem was solved by a brute-force technique that was unlikely to scale to problems larger than a few hundred items. A novel approach to solving this optimization problem is presented. It scales to 20,000 items--the memory limitations of a commodity computational node and within range of problem sizes of biological interest. To further accommodate biological applications, the theory is amended to apply to asymmetric dissimilarity matrices, such as those derived from DNA sequence alignment scores, and the algorithm is extended to recursively examine hierarchical substructure, such as that arising during protein classification.