Effects of the Training Set Size: A Comparison of Standard and Down-Sampled Lexicase Selection in Program Synthesis
Dirk Schweim, Dominik Sobania, Franz Rothlauf · 2022 IEEE Congress on Evolutionary Computation (CEC) · 2022
From a practitioners perspective, the number of in-put/output examples used during the training process in program synthesis studies is too large, as in practice, these examples must be labeled by hand. Therefore, this paper analyzes the influence of different training set sizes on the performance, generalization ability, as well as the structure of the programs generated by grammar-guided genetic programming. We compare down-sampled lexicase selection with standard lexicase selection on three common problems from the general program synthesis benchmark suite. First, we find that both lexicase variants are robust against reducing the amount of training data. We find that standard lexicase has a tendency to overfit the training data on some problems. With down-sampled lexicase, in contrast, overfitting on training data is reduced and evolved programs generalize better on held-out test cases. Consequently, we suggest to use grammar-guided genetic programming with down-sampled lexicase selection in the program synthesis domain.