Using Large Language Models to Mitigate Human-Induced Bias in SMS Spam: An Empirical Approach
Gerard Shu Fuhnwi, Matt Revelle, Bradley M. Whitaker, Clemente I. Izurieta · 2025
Short Message Service (SMS) is a widely used text messaging feature on both basic and smartphones. SMS spam detection is a crucial task. Traditional machine learning approaches often struggle in this domain due to their reliance on manually crafted features, such as keyword detection, which can result in overly simplistic patterns and misclassification of more complex messages. With this shortcoming, these models can amplify human-induced biases if the training data contains inconsistent labeling or subjective interpretations, leading to unfair treatment of specific keywords or contexts. Conversely, advanced LLMs present effective approaches to addressing such issues, as they can more accurately capture linguistic patterns, contextual nuances, and textual ambiguities than traditional models, representing a substantial advancement in improving label accuracy. This paper proposes utilizing LLMs to address humaninduced labeling bias in spam detection and applying different prompt design methods to guide the process. In text classification, we surveyed two leading-edge LLMs, ChatGPT and Gemini, and evaluated them on the English SMS spam dataset source from UC Irvine’s Machine Learning Repository. We explored the highest-performing prompt designs using approaches like in-context learning. The findings indicate that in-context techniques for prompting improve model effectiveness by reducing human-induced (contextual) labeling bias in SMS spam detection with a Balanced Accuracy of 82% $\mathbf{97 \%}$ and an Equal Opportunity Difference (EOD) of precisely zero, indicating LLMs’ trustworthiness (fairness) in reducing this bias compared to traditional machine learning approaches. Our results also suggested that expanding the sample size can decrease LLMs’ ability to reduce human-induced labeling bias in spam detection. In general, this study provides information on the strengths and limitations of LLMs and suggestions for methods to minimize human-induced labeling bias in spam detection and can help guide the selection of appropriate LLMs for this task.