A Large-scale Non-standard English Database and Transformer-based Translation System
Arghya Kundu, Uyen Trang Nguyen · 2023
Natural language processing (NLP) applications face challenges in understanding and processing region or industry specific language. In real-world scenarios, conversational language used in blogs and social media platforms often contains slang and non-standard words (SNSW). These unconventional terms are not typically present in curated datasets used to train NLP models. As a consequence, the performance of these models is negatively affected when they encounter such real-world linguistic variations. In this paper, we introduce SNSW-DB, an automatically curated, regularly updated, large-scale lexical database from crowdsourced dictionaries. The database contains SNSW terms, example sentences, and their standard English synonyms generated by our synonym generator module. We also propose a transformer-based translation system that converts articles with SNSW to standard English. Translated documents enhance readability and utility for human readers, and can improve the performance of downstream NLP tasks.