Statistical significance in inductive learning
Olivier Gascuel, Gilles Caraux · 1992
Abstract, Inductive learning systems search for regularities that therefore be applied with some assurance to an example describe environmental observations, These systems often use which does not belong to the learning set. In other numeri~~l heu~stics to guide this search, The ~ also sele~t words, statistical significance may be used to assess regulantles, which are good, or the best, according to certain that a learnin y learned g Program reall th ' d d d 1 ' , S " , al h 1 " d some mg an 1 numenca cntena, tatlstlc measures ave recent y game t I ' 1 " , th AI ' Th ' "d " 1 no on y provl de ar te fac ts, popu anty m e communIty, IS paper provi es a simp e method for fully exploiting statistical measures, We give a test However, there is a difficulty. Let us continue our which may be used to decide whether a given regularity is previous example. A classification rule provided by a statistically significant, or, in other words, whether this regularity system is found by optimizing a given measure in the may be distinguished from random " This method directly uses learning set. Therefore, the score obtained in the results obta~ed ~rom the learning set with~ut ~qui,ring any test learning set by this rule with this measure, inevitably set. An application to Concep~al Oustenng IS given and,the provides an optimistically biased view of the real performance of the method IS evaluated by a numencal, ul P " all ' d f 1 d d d slm atlon, In y, we provi e Ie erences to Ie ate stu les an, d b tabl t " some e a e ques Ions. performance of the rule, The same holds when CrIterIa wlth related semantIcs are used, e.g. entropic and likelihood ratio criteria in CN2. Moreover, because of the (large) number of possible rules, there are always 1. Introduction some rules which seem good according to the measure, I d t. I ' t h " I ' tI ' th t even when the data are only random. This explains the