Impolite Words Detection Algorithm for Standard Thai and Thai Regional Dialects
Kanida Charungchit, Nannapas Abhiratthanarath, Chouvalit Khancome · 2024
Impolite word detection, also known as rude word detection, is a natural language processing (NLP) technique designed to identify inappropriate or offensive language in texts and various media. This technique aims to prevent the use of language that could be considered inappropriate or cause discomfort in online communities and communication applications. This research article introduces a novel approach for detecting impolite Thai words, encompassing various Thai dialects, and identifying the regional origin of the offensive words. The new algorithm employs multiple dictionaries; each stored using inverted lists based on character positions rather than direct word storage. This dictionary structure enables the search process to achieve 100% efficiency with no errors. Experimental results from a program developed to detect impolite words in standard Thai, as well as Northern, Southern, and Northeastern Thai dialects, demonstrated 100% accuracy, significantly outperforming previous detection algorithms. Additionally, the new algorithm outperformed current Generative AI models, even without considering sentence context or utilizing AI.