Optimizing Indonesian Tweet Preprocessing on Halal Domain
Ekasari Nugraheni, Firhan Imam Haekal, Andria Arisal, Rizal Setya Perdana · 2024
Indonesians' adoption of the halal lifestyle significantly influences the country’s digital landscape, presenting opportunities for data analysis and natural language-based solutions. However, various cases of unstructured and informal Indonesian text pose challenges that standard preprocessing techniques may be inadequate. This study introduces an extended preprocessing method for Indonesian tweets through the module idtext_normalizer, incorporating techniques such as string cleaning, exaggerated letter removal, laughter expression removal, substring separation, duplicate removal based on sequence similarity, and corpus-based filtering. The results demonstrate that these extended preprocessing steps effectively improve the performance of each tested model, and reduce the training time, underscoring the effectiveness of the proposed preprocessing technique in managing unstructured and informal Indonesian language text to optimize model performance.