When and Why Does Bias Mitigation Work?
Abhilasha Ravichander, Joe Stacey, Marek Rei · 2023
Neural models have been shown to exploit shallow surface features to perform language understanding tasks, rather than learning the deeper language understanding and reasoning skills that practitioners desire.Previous work has developed debiasing techniques to pressure models away from 'spurious' features or artifacts in datasets, with the goal of having models instead learn useful, task-relevant representations.However, what do models actually learn as a result of such debiasing procedures?In this work, we evaluate three model debiasing strategies, and through a set of carefully designed tests we show how debiasing can actually increase the model's reliance on hidden biases, instead of learning robust features that help it solve a task.Furthermore, we demonstrate how even debiasing models against all shallow features in a dataset may still not help models address a task.As a result, we suggest that only debiasing existing models may not be sufficient for many language understanding tasks, and future work should consider new learning paradigms to address complex challenges such as commonsense reasoning.