Noise Isn’t Always Negative: Countering Exposure Bias in Sequence-to-Sequence Inflection Models
Garrett Nicolai, Miikka Silfverberg · 2020
Morphological inflection, like many sequence-to-sequence tasks, sees great performance from recurrent neural architectures when data is plentiful, but performance falls off sharply in lowerdata settings.We investigate one aspect of neural seq2seq models that we hypothesize contributes to overfitting -teacher forcing.By creating different training and test conditions, exposure bias increases the likelihood that a system too closely models its training data.Experiments show that teacher-forced models struggle to recover when they enter unknown territory.However, a simple modification to the training algorithm to more closely mimic test conditions creates models that are better able to generalize to unseen environments.