Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-training
Miriam Anschütz, Joshua Oehms, Thomas Wimmer, Bartłomiej Jezierski, Georg Groh · 2023
Automatic text simplification systems help to reduce textual information barriers on the internet.However, for languages other than English, only few parallel data to train these systems exists.We propose a two-step approach to overcome this data scarcity issue.First, we fine-tuned language models on a corpus of German Easy Language, a specific style of German.Then, we used these models as decoders in a sequence-to-sequence simplification task.We show that the language models adapt to the style characteristics of Easy Language and output more accessible texts.Moreover, with the style-specific pre-training, we reduced the number of trainable parameters in text simplification models.Hence, less parallel data is sufficient for training.Our results indicate that pre-training on unaligned data can reduce the required parallel data while improving the performance on downstream tasks.