From apologies to insights: extracting topics from ChatGPT apologetic responses
Brahim Hnich, Ali Ben Mrad, Abdoul Majid O. Thiombiano, Mohamed Wiem Mkaouer · Journal of Decision System · 2025
This article aims to pinpoint areas of suboptimal performance in ChatGPT’s responses, with a specific emphasis on instances containing the term ‘apologies’. Our primary objective is to conduct a nuanced analysis, shedding light on potential shortcomings within ChatGPT’s generated responses and delineating areas requiring improvement. In addition to this primary focus, we introduce an additional layer of analysis, delving into responses where ChatGPT issues apologies. In this context, we aim to identify the underlying tasks or queries that developers present to ChatGPT, revealing specific tasks where the model may demonstrate unreliability, such as refactoring or code completion. Furthermore, employing the Latent Dirichlet Allocation algorithm, we determine topics to gain insights into the most discussed themes with ChatGPT. Using a targeted methodology, we extract key insights to understand the topics where ChatGPT may exhibit deficiencies comprehensively. To validate the apologetic topics further, we adopt an innovative approach leveraging the recent advancement of transformer-based Large Language Models (LLMs). We prompt Claude to assign for each apologetic response, the most appropriate topic among the list of all topics. We compared the topics identified by the LLM model with those determined using LDA and experts. We achieved a consensus rate of 77.5%. This allowed us to gain more confidence in the mined apologetic topics.