On the Use of Emulab Testbeds for Scientifically Rigorous Experiments
Christos Siaterlis, Andres Perez Garcia, Béla Genge · IEEE Communications Surveys & Tutorials · 2012
Internet is considered a Critical Infrastructure (CI) that is vital for both the economy and the society. Disruptions caused by natural disasters, malicious human actions and even hardware failure pose serious risks. Emulation testbeds are increasingly used to study the Internet in order to improve protection and response mechanisms. These are frequently considered more adequate than software simulators to realistically recreate the complex behavior of networks. In this paper we study how testbeds based on the Emulab software can be used to conduct scientifically rigorous experiments, specifically in terms of: a) experiment fidelity, b) repeatability, c) measurement accuracy, and d) interference. Our study, which is based on extensive experimentation on different testbeds, indicates that the current trend of using emulation testbeds is justified as both realistic and efficient. We show that Emulab-based experiments are representative of real systems in terms of emerging behavior (qualitative) and that repeatable experiments are possible. The main contribution of this tutorial article is that based on experimental results we identified caveats and provided insights to significant configuration parameters and limitations that are further elaborated into a set of guidelines that any Emulab user should be aware of. Then, he/she can decide about the importance of each guideline in the context of a specific study and experiment.