Region-dependent temperature scaling for certainty calibration and application to class-imbalanced token classification
Hillary Dawkins, Isar Nejadgholi · 2022
Certainty calibration is an important goal on the path to interpretability and trustworthy AI.Particularly in the context of human-in-theloop systems, high-quality low to mid-range certainty estimates are essential.In the presence of a dominant high-certainty class, for instance the non-entity class in NER problems, existing calibration error measures are completely insensitive to potentially large errors in this certainty region of interest.We introduce a region-balanced calibration error metric that weights all certainty regions equally.When low and mid certainty estimates are taken into account, calibration error is typically larger than previously reported.We introduce a simple extension of temperature scaling, requiring no additional computation, that can reduce both traditional and region-balanced notions of calibration error over existing baselines.