Comprehensive Hausa Language Processing Models for Text Summarization, Sentiment Analysis, Machine Translation, and Question Answering
Kabiru Uba Kiru, Surya Kant Pal, Khalil Haruna Aminu, Rita Roy · 2024
Natural Language Processing (NLP) is one of the main streams of AI, making it possible for machines to communicate in a human language, whether written or spoken. Despite significant research and rapid advancements in NLP, many less-resourced languages remain underserved. This project addresses this gap by developing customized and robust NLP models to support four major tasks: text summarization, sentiment analysis, machine translation (Hausa-English and English-Hausa), and question answering. The study uses advanced machine-learning approaches and pre-trained models to make the Hausa language more accessible and useful to the Hausa-speaking community and lay a foundation for integrating the Hausa language into the greater AI ecosystem. Furthermore, feature engineering, data augmentation, and transformation help to overcome the scarcity of Hausa language resources by generating a diverse and heterogeneous dataset. Several pre-trained models, including BART and PEGASUS for summarization, MarianMT for machine translation, and BERT for question answering, were fine-tuned for Hausa-specific tasks; also, traditional and advanced machine learning techniques employed in sentiment analysis, such as LSTM, CNN, SVM, and Random Forest, were applied, with CountVectorizer for text vectorization. The HuggingFace Trainer class was used for training and evaluation. Results show exceptional performance across all models, with BART and PEGASUS notably producing high-quality summaries, MarianMT enabling accurate translations between Hausa and English, and BERT delivering reliable question-answering results. Sentiment analysis models also showed solid performance based on accuracy, precision, recall, and F1 scores. Upon successful development, this study will open new opportunities for Hausa speakers to access digital information more easily. It highlights the importance of integrating remnant languages into the global AI ecosystem, setting the stage for future NLP developments in less-resourced languages.