Empirical Assessment of Nonconformity Score Functions for Classifiers in Conformal Prediction
Bhargava Kumar, Tejaswini Kumar, Swapna Nadakuditi · International Journal of Computer Trends and Technology · 2024
Conformal prediction provides statistically guaranteed confidence measures for any machine learning model. This study investigates the effectiveness of three non-conformity score functions, namely Adaptive Prediction Sets (APS), Regularized Adaptive Prediction Sets (RAPS), and Sorted Adaptive Prediction Sets (SAPS), for sentiment analysis tasks. Expanding on past research that demonstrated the superiority of SAPS in classification tasks for image data, this study assesses whether this superiority extends to other domains, such as sentiment analysis. The study aims to evaluate these non-conformity score functions based on coverage and set sizes. The researchers conducted extensive experiments on a sentiment classification task using the GoEmotions dataset to gain insights into the versatility of SAPS and compared its performance with APS and RAPS. By examining the effectiveness of these non-conformity score functions, this study contributes to the understanding of the practicality of conformal prediction methods in real-world machine learning tasks beyond image classification.