StatVis: A visual analytics framework for statistical cluster validation in high dimensions
Donia Y. Badawood · Array · 2025
High-dimensional datasets are common across scientific and industrial domains, yet interpreting their clustering outcomes remains a complex challenge due to projection distortions and metric ambiguities. Traditional visualization techniques, such as Principal Component Analysis (PCA) or t-Distributed Stochastic Neighbor Embedding (t-SNE), often obscure critical information about cluster validity, separability, and density. This paper introduces StatVis, a statistical manifold learning framework designed to enhance Visual Cluster Validation (VCV) in high-dimensional spaces. The framework combines Dimensionality Reduction (DR) with internal validation metrics and density estimation to generate interpretable, statistically grounded cluster visualizations. StatVis integrates manifold learning techniques (Uniform Manifold Approximation and Projection (UMAP), t-SNE) with clustering algorithms (k-means) and calculates multiple validation metrics, including the Silhouette Coefficient, Davies–Bouldin Index, and Dunn Index. Local densities are estimated using both kernel density estimation and k-nearest neighbor methods. Experiments were conducted on three standard datasets: the UCI Dry Bean, MNIST, and 20 Newsgroups datasets. Visual overlays, projection distortion maps, and density-aware correction routines were evaluated. StatVis demonstrated superior capability in identifying and visually communicating cluster quality and projection inconsistencies. It outperformed standard visualizations in conveying cluster separability and compactness. The density-aware correction routine further improved interpretability in regions of high distortion. StatVis effectively bridges statistical validation and visualization, enabling more reliable and explainable cluster analysis. The framework shows promise as a visual decision support tool for exploring high-dimensional data and incorporating human input into machine learning.