Navigating the Labyrinth: A Review of Explainability and Trustworthiness in Large Language Model-Powered Systems for Sensitive Decision-Making
Ayman Asaad, A. M. Azizul Hassan Chy, Anzam Shahriar Kabir, Amrun Nakib, Nazifa Tabassum · Scientia. Technology, science and society. · 2025
Large Language Models (LLMs) have proven to be game-changers in sectors that require high-stakes decision-making, such as healthcare, finance, and law. Despite this, LLMs are opaque, black boxes that present significant challenges in terms of explainability and trust, particularly when ethical, legal, and regulatory requirements necessitate transparency. This review examines the state-of-the-art in the intersection of Large Language Models (LLMs), Explainable Artificial Intelligence (XAI), and trustworthy AI. It addresses four research questions: the faithfulness of LLMs' explanations, the current state of evaluation, user-centered and ethical design in XAI systems, and the broader challenge of making LLMs themselves explainable. Using a systematic literature review approach, we reviewed over 500 sources published between 2018 and 2025 and narrowed our selection to draw insights from 40 significant contributions. We categorized the XAI methods into post-hoc, intrinsic, and human-centered approaches, and then mapped these methods to the categories of trustworthiness, including robustness, fairness, privacy, and accountability. The review revealed recurrent themes, including the hallucination of LLM explanations, limited evaluation metrics, the urgent need to create interactive explanation interfaces, and the meta-challenge of explaining complicated LLMs. As a consequence of the review process, we propose a research agenda with short-, medium-, and long-term priorities, including faithfulness verification, user modeling, symbolic-LLM hybrid architectures, and LLMs that are interpretable by design. This review synthesis contributes to the corpus of technical, ethical, and context-specific knowledge, offering foundational insights into the design of transparent, reliable, and human-aligned LLM-powered systems for sensitive domains.