Distributionally Robust Finetuning BERT for Covariate Drift in Spoken Language Understanding
Samuel Broscheit, Quynh Do, Judith Gaspers · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2022
In this study, we investigate robustness against covariate drift in spoken language understanding (SLU).Covariate drift can occur in SLU when there is a drift between training and testing regarding what users request or how they request it.To study this we propose a method that exploits natural variations in data to create a covariate drift in SLU datasets.Experiments show that a state-of-the-art BERT-based model suffers performance loss under this drift.To mitigate the performance loss, we investigate distributionally robust optimization (DRO) for finetuning BERT-based models.We discuss some recent DRO methods, propose two new variants and empirically show that DRO improves robustness under drift.