Canonical Segmentation Using Affix Characters as a Unit on Transformer for Javanese Language
Sri Hartati Wijono, Machmud Roby Alhamidi, Muhammad Hafizhuddin Hilman, Wisnu Jatmiko · 2021
Morphological segmentation for agglutinative languages is the process of getting stems and affixes. Morphological segmentation is a necessary process in various NLP applications such as machine translation, question answering, and speech recognition. Several neural morphological segmentation studies have used the sequence of characters as input to encoder-decoder. However, this can not provide linguistic information. We propose affix characters as a unit to provide affixes feature on Transformer encoder-decoder. We use the Javanese word corpus which consists of affixed, canonical affixed, and non-affixed words. For affixed words, our proposed method obtains 11.2 times higher point of accuracy than the Sequence of Characters. For canonical affixed words, we get 21.9 times higher point of accuracy than the baseline method. The results also show that the use of different affix symbols, which are “%%”, “##”, and “@@” for each type of affix improve accuracy in affix recognition.