BP: DECREE: A Platform for Repeatable and Reproducible Security Experiments
Lok Kwong Yan, Benjamin Price, Michael Zhivich, Brian Caswell, Christopher S. Eagle, Michael Frantzen, Holt Sorenson, Michael Thompson, Timothy M. Vidas, Jason T. Wright, Vernon Rivet, Samuel Colt VanWinkle, Clark Wood · 2018
Reproducibility is one of the core tenets of science. Researchers must be able to independently verify scientific theories and research results; practitioners need to able to trust test results and assessments. Reproducibility is hard because computing environments can exhibit non-deterministic and even diverging behaviors due to differing hardware and software configurations. Software running on one platform might behave differently when executed on another. One might think that constraining configurations using virtualization is the answer, but even machine instructions can be sources of non-determinism. The challenges, trade-offs, lessons learned and our solutions (e.g., use a restricted instruction set) in creating DECREE, a repeatable and reproducible computing environment are detailed in this paper.