ASOS at NADI 2024 shared task: Bridging Dialectness Estimation and MSA Machine Translation for Arabic Language Enhancement
Omer Nacar, Serry Sibaee, Abdullah Alharbi, Lahouari Ghouti, Anis Koubâa · 2024
This study undertakes a comprehensive investigation of transformer-based models to advance Arabic language processing, focusing on two pivotal aspects: the estimation of Arabic Level of Dialectness and dialectal sentencelevel machine translation into Modern Standard Arabic.We conducted various evaluations of different sentence transformers across a proposed regression model, showing that the MARBERT transformer-based proposed regression model achieved the best root mean square error of 0.1403 for Arabic Level of Dialectness estimation.In parallel, we developed bi-directional translation models between Modern Standard Arabic and four specific Arabic dialects-Egyptian, Emirati, Jordanian, and Palestinian-by fine-tuning and evaluating different sequence-to-sequence transformers.This approach significantly improved translation quality, achieving a BLEU score of 0.1713.We also enhanced our evaluation capabilities by integrating MSA predictions from the machine translation model into our Arabic Level of Dialectness estimation framework, forming a comprehensive pipeline that not only demonstrates the effectiveness of our methodologies but also establishes a new benchmark in the deployment of advanced Arabic NLP technologies.