A new method to compute classification error
Abhranil Das, R. Calen Walshe, Wilson S. Geisler · Journal of Vision · 2019
The sensitivity index d’ is used ubiquitously to express the error rate in classification and detection tasks. In the general case with multiple cues that can have arbitrary variances and pairwise correlations, and unequal prior class probabilities, the decision boundary separating the classes is a complex surface. This makes it challenging to compute the error rate integral, and there currently exists no single algorithm that works reliably in all cases. Standard integration procedures may require grids that are inefficiently large and fine, converge slowly, yet may miss relevant parts of the space unless tailored case-by-case. These obstacles impede the testing and optimization of models. We present a new method to compute error rates simply, reliably and fast, for all cases up to 3 dimensions. This is founded on a mathematical transformation of the feature space that converts all cases into a single canonical form that is simple and spherically symmetric. In polar coordinates that can exploit this symmetry, the radius integral has a known analytical form, and the angle integral is computed over the same bounded grid of angles, for all cases. Our open-source MATLAB function based on this method computes error rates and d’, returns the optimal decision boundary, accommodates custom suboptimal boundaries, and produces plots to visualize the distributions and the decision boundary. We have applied this method to compute error rates for an ideal observer analysis of a detection task in natural scenes. This was a particularly challenging case for a traditional integration method because: i.) there are three correlated cues used for detection, ii.) error rates are extremely small (below machine precision) in some conditions, and iii.) the decision boundary varies unpredictably across conditions. We find that this method recovers the true error rate quickly and accurately, and to an arbitrary level of precision.