A Combined Approach to Performance Regression Testing Resource Usage Reduction
Milad Abdullah, David Georg Reichelt, Vojtěch Horký, Lubomír Bulej, Tomáš Bureš, Petr Tůma · 2025
Performance regression testing is often seen as a natural part of the continuous integration pipeline. The underpinning layers, such as just-in-time compilation, memory mapping, and operating system characteristics, often influence performance measurement samples. To reduce such non-deterministic factors, the usual practice includes restarting the measured workload, performing warmups, and controlling environmental variability. These need to be parameterized, among others, by run count, warm-up iterations, and iteration count. Importantly, performance testing that detects performance regressions of any scale is computationally expensive due to the need to collect samples that can detect performance changes with statistical significance.