A Robust Backward Compatibility Metric for Model Retraining
Ryuta Matsuno, Keita Sakuma · 2023
Model retraining and updating are essential processes in AI applications. However, during updates, there is a potential for performance degradation, in which the overall performance improves, but local performance deteriorates. This study proposes a backward compatibility metric that focuses on the compatibility of local predictive performance. The score of the proposed metric increases if the accuracy over the conditional distribution for each input is higher than before. Furthermore, we propose a model retraining method based on the proposed metric. Due to the use of the conditional distribution, our metric and retraining method are robust against label noises, while existing sample-based backward compatibility metrics are often affected by noise. We perform a theoretical analysis of our method and derive an upper bound for the generalization error. Numerical experiments demonstrate that our retraining method enhances compatibility while achieving equal or better trade-offs in overall performance compared to existing methods.