Truecluster: scalable statistical clustering with model selection
Jens Oehlschlägel · arXiv (Cornell University) · 2006
Data based classification is fundamental to most branches of science. Despite of progress in statistical computing and predictive modelling, cluster analysis until today lacks model selection robustness and scalability to large datasets. We consider the important problem of deciding about the optimal number of clusters given an arbitrary definition of space and clusteriness. We show how to construct a Cluster Information Criterion that allows objective model selection. Differing from other approaches, our truecluster method does not require specific assumptions about underlying distributions, distance definitions or cluster models. Truecluster puts arbitrary clustering algorithms into a generic unified (sampling based) statistical framework. It is scalable to big datasets and provides robust cluster assignments and casewise diagnostics. Truecluster will make clustering more objective, allows for automation and will save time and costs. ∗ www.truecluster.com Copyright (C) Dr. Jens Oehlschlägel 2005, all rights reserved. We thank Thomas Augustin and Stefan Pilz for their helpful comments on the draft of this paper. 1