Simple Measures of Individual Cluster-Membership Certainty for Hard Partitional Clustering
Dongmeng Liu, Jinko Graham · The American Statistician · 2018
Dongmeng Liua & Jinko Grahama* a Department of Statistics and Actuarial Science, Simon Fraser University, Burnaby, BC, CanadaCONTACT Jinko Graham [email protected] Department of Statistics and Actuarial Science, 8888 University Drive, Simon Fraser University, Burnaby, BC, Canada V5A 1S6This article did not complete review and production in time for inclusion in The American Statistician's special Issue on Data Science (February 2018).ABSTRACTWe propose two probability-like measures of individual cluster-membership certainty that can be applied to a hard partition of the sample such as that obtained from the partitioning around medoids (PAM) algorithm, hierarchical clustering or k-means clustering. One measure extends the individual silhouette widths and the other is obtained directly from the pairwise dissimilarities in the sample. Unlike the classic silhouette, however, the measures behave like probabilities and can be used to investigate an individual’s tendency to belong to a cluster. We also suggest two possible ways to evaluate the hard partition using these measures. We evaluate the performance of both measures in individuals with ambiguous cluster membership, using simulated binary datasets that have been partitioned by the PAM algorithm or continuous datasets that have been partitioned by hierarchical clustering and k-means clustering. For comparison, we also present results from soft-clustering algorithms such as soft analysis clustering (FANNY) and two model-based clustering methods. Our proposed measures perform comparably to the posterior probability estimators from either FANNY or the model-based clustering methods. We also illustrate the proposed measures by applying them to Fisher’s classic dataset on irises.