Feature analysis for proper intensity scaling and feature distinction in class activation maps
Yanli Li, Denis P. Shamonin, Tahereh Hassanzadeh, M. Reijnierse, Annette H M van der Helm–van Mil, Berend C. Stoel · Knowledge-Based Systems · 2025
Understanding the decisions of deep learning (DL) models is crucial for their acceptance in risk-sensitive applications. Class activation maps (CAMs) are commonly used in image analysis to visualize model reasoning by generating attention maps where signal intensities represent contributions to outputs. However, existing CAM algorithms focus on optimal weight design and salient feature layer selection, neglecting two key limitations in population-level interpretation: (1) lack of an appropriate intensity scale for proper interpretation and quantitative analysis, and (2) inability to identify which features within selected layers predominantly drive model reasoning. These gaps can lead to miscorrelations between CAMs and model outputs, causing erroneous interpretations, while restricting CAMs to case-specific visual inspections. We propose a framework to statistically analyze DL-extracted features at a population level, determinizing feature contributions for global intensity scaling and within-layer feature distinction. The global intensity scale standardizes CAMs, achieving high correlation coefficients (R) with model outputs. Within-layer feature distinction identifies overfitting, confounding factors, outliers, redundancies, and principal features. Applied to eight datasets, including five medical imaging datasets, this framework improved Rs between CAMs and outputs by 10.7–64.2%, achieving near 100% consistency. Furthermore, distinguishing principal features (5–25% in the selected layer) produced CAMs of equal quality while maintaining model output accuracy. This method, relying on minimal assumptions, enhances CAM-model consistency and broadens CAM applications by enabling standardized interpretation and deeper feature-level insights.