Making Decision Support Systems Intelligible: Explainability or Interpretability?
Alessandra Buccella, Uri Maoz · 2025
Artificial intelligence (AI) systems assist humans to predict future outcomes by analyzing large, heterogeneous data of past patterns. In particular, Decision Support Systems (DSSs) leverage AI to analyze vast amounts of information and provide insights quickly, making them particularly attractive to professionals who routinely make high-stakes decisions under time pressure. DSSs are used in various fields—including medical diagnostics, insurance, and criminal justice—and are increasingly incorporating Large Language Models (LLMs). But for AI-powered DSSs to be effective in assisting professionals, their outputs as well as inner workings must be intelligible. In this paper, we argue that “Interpretable” AI models, i.e. models whose inner workings are simple enough for humans to comprehend, are better suited to provide the understanding users need compared to more complex and opaque models, whose outputs are ‘explained’ through so-called “Explainable AI” (XAI) tools. Professionals need to be able to communicate the reasons for their decisions to patients, clients, etc., and while explanations constructed through XAI techniques do not necessarily provide users with the kind of information required for reason-giving, interpretable models do.