On uncertainty measures used for decision tree induction
Louis A. Wehenkel · Open Repository and Bibliography (University of Liège) · 1996
This paper provides a further look at uncertainty or information criteria used in the context of decision tree induction, and more generally of learning conditional class probability models. We show the high degree of similarity among two main families of criteria based respectively on the logarithmic SHANNON entropy function and the quadratic GINI index. We start by introducing a general family of entropy functions and then discuss the latter particular cases, and end up with a short review of the Kolmogorov-Smirnov distance, another related measure. 1 Generalized Information Functions The concept of generalized information functions of type β was first introduced by Daróczy [1] and its use for pattern recognition problems was discussed by Devijver [2]. The entropy of type β (β positive and different from 1) of a discrete probability distribution (p1,..., pm) is defined by m� piu β (pi), (1) H β (p1,..., pm) △ = i=1 where the uncertainty u β (pi) is defined by u β (pi) △ = 2β−1 (1 − pβ−1 i). (2) 2β−1 − 1 Measure uβ is a strictly decreasing function of pi.