Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems
Jindřich Libovický, Alexander Fraser · 2020
Applying the Transformer architecture on the character level usually requires very deep architectures that are difficult and slow to train.These problems can be partially overcome by incorporating a segmentation into tokens in the model.We show that by initially training a subword model and then finetuning it on characters, we can obtain a neural machine translation model that works at the character level without requiring token segmentation.We use only the vanilla 6-layer Transformer Base architecture.Our character-level models better capture morphological phenomena and show more robustness to noise at the expense of somewhat worse overall translation quality.Our study is a significant step towards highperformance and easy to train character-based models that are not extremely large.