On learning and testing evaluation functions
Bruce D. Abramson · Journal of Experimental & Theoretical Artificial Intelligence · 1990
. Artificial intelligence (AI) is. in many respects, an experimental science. Nevertheless, little has been written about the applicability of the scientific method to AI. The literature concerning techniques for learning and testing static evaluators provides a case in point: experiments are widely discussed, but rarely analyzed. These experiments have played prominent roles in both heuristic search and machine learning; in general, progressively stronger evaluators are learned and competitive abilities are tested. There are two basic problems with this approach. First, learning and improvement are not synonymous; there have been few attempts to specify precisely what is being learned. Second, competition blurs a program' s constituent components into a single performance measure. An alternative approach would be to identify a key parameter, to learn evaluators that approximate it, and then to challenge otherwise identical programs with marginally different functions. This paper reviews some previous work in respect to its adherence to the general principles of experiment design. It also describes experiments that learn coefficients for chess material-advantage functions that approximate expected outcome, and then describes a tournament involving expert-design, learned and randomly generated material-advantage evaluators. The results of this tournament challenge many long-held beliefs and stress the need for greater precision in experimental AI.