Exploring the Hidden Challenges Associated with the Evaluation of Multi-class Datasets Using Multiple Classifiers
Shamaila Iram, Dhiya Al‐Jumeily, Paul Fergus, Abir Jaafar Hussain · 2014
The optimization and evaluation of a pattern recognition system requires different problems like multi-class and imbalanced datasets be addressed. This paper presents the classification of multi-class datasets which present more challenges when compare to binary class datasets in machine learning. Furthermore, it argues that the performance evaluation of a classification model for multi-class imbalanced datasets in terms of simple "accuracy rate" can possibly provide misleading results. Other parameters such as failure avoidance, true identification of positive and negative instances of a class and class discrimination are also very important. We, in this paper, hypothesize that "misclassification of true positive patterns should not necessarily be categorized as false negative while evaluating a classifier for multi-class datasets", a common practice that has been observed in the existing literature. In order to address these hidden challenges for the generalization of a particular classifier, several evaluation metrics are compared for a multi-class dataset with four classes, three of them belong to different neurodegenerative diseases and one to control subjects. Three classifiers, linear discriminant, quadratic discriminant and Parzen are selected to demonstrate the results with examples.