Necessary but not Sufficient: Limitations of Projection Quality Metrics
Alister Machado, Michael Behrisch, Alexandru Cristian Telea · Computer Graphics Forum · 2025
Abstract High‐dimensional data analysis often uses dimensionality reduction (DR, also called projection) to map data patterns to human‐digestible visual patterns in a 2D scatterplot. Yet, DR methods may fail to show true data patterns and/or create visual patterns that do not represent any data patterns. Projection Quality Metrics (PQMs) are used as objective measures to gauge the above process: the higher a projection's scores in PQMs, the more it is deemed faithful to the data it represents. We show that, while PQMs can be used as exclusion criteria — low values usually mean poor projections — the converse does not always hold. For this, we develop a technique to automatically generate projections that score similar or even higher PQM values than projections created by well‐known techniques, but show different, often confusing, visual patterns. Our results show that accepted PQMs cannot be used as an exclusive way to tell whether a projection yields accurate and interpretable visual patterns — in this sense, PQMs play a role akin to that of summary statistics in exploratory data analysis. We also show that not all studied metrics can befooled equally well, suggesting a ranking of metrics in their ability to reliably capture quality.