When Usability Undermines Explainability: A Case Study Designing AI-Infused Technologies in Healthcare
Mengyun Li, Dane A. Morey, Akhil Modali, Jacqueline Brandon, Michael F. Rayo · Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care · 2025
Designing AI-infused technologies in healthcare involves multidisciplinary design considerations, like the usability of the display and explainability of the AI algorithm. We present three vignettes from a case study in which design decisions intended to improve usability inadvertently undermined explainability, and vice versa. Our results illustrate potential conflicts between disciplines and suggest the need for joint activity as a unifying design perspective. From these insights, we discuss revisions to our AI-infused technology.