The RERS challenge: towards controllable and scalable benchmark synthesis
Falk Maria Howar, Marc Jasper, Malte Mues, David Maria Schmidt, Bernhard Steffen · International Journal on Software Tools for Technology Transfer · 2021
Abstract This paper (1) summarizes the history of the RERS challenge for the analysis and verification of reactive systems, its profile and intentions, its relation to other competitions, and, in particular, its evolution due to the feedback of participants, and (2) presents the most recent development concerning the synthesis of hard benchmark problems. In particular, the second part proposes a way to tailor benchmarks according to the depths to which programs have to be investigated in order to find all errors. This gives benchmark designers a method to challenge contributors that try to perform well by excessive guessing.