Improving Character-Aware Neural Language Model byWarming Up Character Encoder under Skip-gram Architecture
Yukun Feng, Chenlong Hu, Hidetaka Kamigaito, Hiroya Takamura, Manabu Okumura · 2021
Character-aware neural language models can capture the relationship between words by exploiting character-level information and are particularly effective for languages with rich morphology.However, these models are usually biased towards information from surface forms.To alleviate this problem, we propose a simple and effective method to improve a character-aware neural language model by forcing a character encoder to produce wordbased embeddings under Skip-gram architecture in a warm-up step without extra training data.We empirically show that the resulting character-aware neural language model achieves obvious improvements of perplexity scores on typologically diverse languages, that contain many low-frequency or unseen words.