Benchmark Generation with VEVOS: A Coverage Analysis of Evolution Scenarios in Variant-Rich Systems
Alexander Schultheiß, Paul Maximilian Bittner, Sandra Greiner, Timo Kehrer · 2023
Clone-and-own development is a simple and flexible approach to realize multi-variant software systems in practice but typically provokes costly challenges in maintaining a continuously evolving set of variants. Therefore, managed clone-and-own development is key to efficiently mitigate these problems. While supporting techniques have been proposed in the literature, hardly any of them have been evaluated in a realistic setting due to a substantial lack of publicly available clone-and-own projects which could be used as experimental subjects. Recently, we presented the benchmark generation framework VEVOS for simulating clone-and-own development. However, it is yet unclear to which extent VEVOS can cover key scenarios for evaluating evolving variant-rich systems. This paper examines to what extent benchmarks created by VEVOS satisfy evaluation requirements for evolution scenarios demanded by the community. In addition, we report on our own experiences when employing VEVOS within six studies and elaborate on necessary extensions we implemented into VEVOS.