Automatic Dialect Detection for Low Resource Santali Language
Sunil Kumar Sahoo, Brojo Kishore Mishra, Shantipriya Parida, Satya Ranjan Dash, Jatindra Nath Besra, Esaú Villatoro-Tello · 2021
Among different natural language processing (NLP) tasks, language detection is considered a difficult task especially for similar languages, varieties, and dialects. To build any tools or technologies in NLP, it is required reliable and robust language detection tool. Although such language detection tools and technologies are available for many high-resource languages, it is difficult to find them for low-resource languages due to the lack of training datasets. In this work, we compiled a language detection dataset for the low resource language Santali. We considered the Sanatali written in Odia script and standard Odia script. We used a deep supervised autoencoder for language detection. The deep supervised autoencoder can detect both Odia and Santali languages although written using the same Odia script. We obtained an accuracy of 100% on the test set. Our proposed dataset will be enriched Santali language and help researchers to build tools and technologies for this low resource language.