Evolution vs. Intelligent Design in Program Patching
Prémkumar Dévanbu · eScholarship (California Digital Library) · 2013
While fixing bugs requires significant manual effort, recent research has shown that genetic programming (GP) can be used to search through a space of programs to automatically find candidate bugfixing patches.Given a program, and a set of test cases (some of which fail), a GP-based repair technique evolves a patch or a patched program using program mutation and selection operators.We evaluate GenProg, a well-known GP-based patch generator, using a large, diverse dataset of over a thousand simple (both buggy and correct) student-written homework programs, using two different test sets: a white-box test set constructed to achieve edge coverage on an oracle program, and a black-box test set developed to exercise the desired specification.We find that GenProg often succeeds at finding a patch that will cause student programs to pass supplied white-box test cases; however, that the solution quite often overfits to the supplied tests and doesn't pass all the black-box tests.In contrast, when students patch their own buggy programs, these patches tend to pass the black-box tests as well.We also find that the GenProg-generated patches lack enough diversity to benefit from a kind of bagging, in which a plurality vote over a population of GP-generated patches outperforms a randomly chosen individual patch.We report these results and additional relationships between GenProg's success and the size and complexity of the manual and automatic patches.