An Unsupervised Text Normalization Architecture for Turkish Language
Savaş Yıldırım, Tuğba Yıldız · Research in Computing Science · 2015
A variety of applications on the problem of short-text messages require text normalization process that transforms ill-formed words into standard ones.Recently, many successful approaches have been applied to text normalization especially for social media text.Since each natural language has its own difficulties and barriers, we need to design an architecture to normalize short text messages in Turkish language which has an morphologically rich agglutinative structure.The model proceeds from simple solutions towards more complicated and sophisticated ones to reduce time complexity.A variety of techniques from lexical similarity to n-gram language modeling have been evaluated by exploiting several resources such as high quality corpus, morphological parser and dictionaries.We demonstrate that unsupervised text normalization architecture adapting both lexical and semantic similarity for Turkish domain has shown efficient results that might contribute to other studies.