Estimating concept difficulty with cross-entropy
Khalil Wafi Zainab Nazar, Max Bramer · 1998
Learning difficulty can vary considerably from one algorithm to another because different approaches may be biased or tuned towards a certain initial problem description. Reasons for poor performance include noise, class distribution, number of attributes or examples in the sample. However, when intrinsic accuracy is high and performance is poor, the problem can be caused by feature interaction. Patterns are more difficult to identify because they are conditional. Systems that attempt to learn in domains such as this can perform constructive induction to change the initial representation to one which makes classification information more visible. However, systems that attempt to reformulate example descriptions often do so regardless of the initial representation. The authors present a data-based detection measure that estimates concept difficulty. Several measures including /spl mu/-ness, variation, blurring (/spl Delta/) and /spl Delta//sub j/ are compared. They argue that a measure based only on the a posteriori probability of the class variable has limited use, and that disparities between concept difficulty and blurring results on some data sets can be explained by employing a simple technique that averages the blurring measure over subsets generated by splitting on the best attribute.