Meta-Learning and Novelty Detection for Machine Learning with Reject Option
Patricia Drapal, Telmo Silva-Filho, Ricardo B. C. Prudêncio · 2024
Preventing a Machine Learning (ML) predictor from making an unreliable prediction is extremely important in sensitive application domains, such as health contexts. In this sense, strategies based on the reject option have been increasingly explored. However, few studies explore the ability of meta-learning to inspect the errors of a base predictor under analysis, in such a way to generalize when the predictor is confident or not. Therefore, the current paper proposes a novel solution for ML with reject option based on the combination of meta-learning and novelty detection. The proposal addresses two distinct situations where a prediction should be rejected. First, novel detection is adopted to identify out-of-distribution instances, i.e., instances that significantly differ from those ones adopted to train the base predictor. Second, meta-learning is adopted to detect instances in regions of data where the base model has shown poor predictive performance during its evaluation. Such instances mainly lie in areas of class overlap or noisy regions in the training data. The results in experiments on synthetic and real data showed the superiority of the solution compared to those based only on meta-learning (aka without novelty detection) and those based on classifier confidence.