Addressing Performance Inconsistency in Domain Generalization for Image Classification
Jamie Stirling, Noura Al Moubayed · 2023
Domain Generalization (DG) in computer vision aims to replicate the human ability to generalize well under a shift of data distribution, or domain. In recent years, the field of domain generalization has seen a steady increase in average left-out test accuracy, measured as the average test accuracy achieved when each domain (in turn) is left out of training and used only for testing. To date, average left-out test accuracy is the only metric used for evaluating and comparing different techniques in DG. We observe that despite the steady increase in average left-out test accuracy, there remains a vast inconsistency between the left-out test accuracy scores measured for individual domains. To the best of our knowledge, this domain inconsistency persists across all published DG methods to date. In this work, we argue that domain generalization cannot be said to be successful without substantially reducing this performance inconsistency between domains. We propose a formal metric for measuring domain inconsistency and apply it to results in the literature. We run experiments to explore how alternative choices of pretraining affects domain inconsistency, finding that, in some settings, careful choice of pretraining can improve consistency with minimal negative (and sometimes positive) impact on average left-out test accuracy. Finally we discuss other potential sources of domain inconsistency and limitations of our work.