Benchmarking Flaws Undermine Security Research
Erik van der Kouwe, Gernot Heiser, Dennis Andriesse, Herbert Bos, Cristiano Giuffrida · IEEE Security & Privacy · 2020
Benchmarking systems is difficult. Mistakes can compromise guarantees and threaten reproducibility and comparability. We conduct a study to show that benchmarking flaws are widespread in systems security defense papers, even at tier-1 venues. We aim to raise awareness and provide recommendations for safeguarding the scientific process in our community.