Workshop on Automatic Machine Learning, AutoML 2016, co-located with 33rd International Conference on Machine Learning (ICML 2016), New York City, NY, USA, June 24, 2016
TU/e Research Portal · 2016
One of the simplest metalearning methods is the average ranking method.This method uses metadata in the form of test results of a given set of algorithms on a given set of datasets and calculates an average rank for each algorithm.The ranks are used to construct the average ranking.We investigate the problem of how the process of generating the average ranking is affected by incomplete metadata involving fewer test results.This is relevant, as such situations are often encountered in practice.In this paper we describe a relatively simple average ranking method that is capable of dealing with incomplete metadata.Our results show that the proposed method is relatively robust to omissions in the test results.This finding could be of use in a future design of experiments.As the incomplete metadata does not affect the final results much, we can simply conduct fewer tests and thus save computation time.