XAI-Eval: A framework for comparative evaluation of explanation methods in healthcare
Krish Agrawal, Radwa El Shawi, Nada Mohamed Osman Sid Ahmed · Digital Health · 2025
Objective: Machine learning systems are increasingly used in high-stakes domains such as healthcare, where predictive accuracy must be accompanied by explainability to ensure trust, validation, and regulatory compliance. This study aims to evaluate the effectiveness of widely used local and global explanation methods in real-world clinical settings. Methods: We introduce a structured evaluation methodology for the quantitative comparison of explainability techniques. Our analysis covers five local model-agnostic methods-local interpretable model-agnostic explanations (LIME), contextual importance and utility, RuleFit, RuleMatrix, and Anchor-assessed using multiple explainability criteria. For global interpretability, we consider LIME, Anchor, RuleFit, and RuleMatrix. Experiments are conducted on diverse healthcare datasets and tasks to assess performance. Results: The results show that RuleFit and RuleMatrix consistently provide robust and interpretable global explanations across tasks. Local methods show varying performance depending on the evaluation dimension and dataset. Our findings highlight important trade-offs between fidelity, stability, and complexity, offering critical insights into method suitability for clinical applications. Conclusion: This work provides a practical framework for systematically assessing explanation methods in healthcare. It offers actionable guidance for selecting appropriate and trustworthy techniques, supporting safe and transparent deployment of machine learning models in sensitive, real-world environments.