Clustering using Renyi's entropy
Robert Jenssen, Kenneth E. Hild, Deniz Erdoğmuş, José Carlos Príncipe, Torbjørn Eltoft · 2004
We propose a new clustering algorithm using Renyi's entropy as our similarity metric. The main idea is to assign a data pattern to the cluster, which among all possible clusters, increases its within-cluster entropy the least, upon inclusion of the pattern. We refer to this procedure as differential entropy clustering. Not knowing the true number of clusters in advance, initially a number of small clusters are "seeded" randomly in the data set, labeling a small subset of the data. Thereafter all remaining patterns are labeled by differential entropy clustering. Subsequently, we identify the "worst cluster" by a quantity we name as the between-cluster entropy. Its members are re-clustered, again by differential entropy clustering, reducing the overall number of clusters by one. This procedure is repeated until only two clusters remain. At each step we store the current labels, thus producing a hierarchy of clusters. The between-cluster entropy also enables us to select our final set of clusters in other cluster hierarchy. We demonstrate the clustering algorithm when applied both to artificially created data sets and a real data set.