Orchid: Building Dynamic Test Oracles with Training Bias for Improving Deep Neural Network Models
Haipeng Wang, W. K. Chan · 2021 8th International Conference on Dependable Systems and Their Applications (DSA) · 2021
The accuracy of deep neural network models is always a top priority in developing these models. One problem to affect it is to what extent such a model can resolve training samples conflicting with one another while learning from them. Such a model may easily learn the whole training dataset with high recall, leaving the problem only to appear when evaluating the model, and yet samples in the evaluation should not be used in training the model. This paper proposes Orchid, the first work to alleviate the problem of class pairs with training samples conflicting with one another. Orchid firstly rewinds the learning effect of a deep neural network model under fix on different subsets of the training dataset, producing a set of resultant rewound models. It then constructs a novel kind of reference oracle by dynamic selection and integration of these rewound models, each predicting the same sample in question correctly. It finally adapts the model under fix by retraining it on training samples with the reference oracle. The experiment on MobileNet and ShuffletV2 with the Cifar-100 and Tiny-ImageNet datasets shows that their test accuracies are improved over the baselines by 2.47% and 2.62%, respectively. Orchid also produces noticeable reductions in misclassification between classes of the adapted models on the test datasets.