Parameterized Unit Testing in the Open Source Wild
Wing Lam, Siwakorn Srisakaokul, Blake Bassett, Peyman Mahdian, Tao Xie, Nikolai Tillmann, Jonathan de Halleux · 2015
With recent advances in test generation research, powerful test generation tools are now at the ngertips of developers in software industry. For example, Microsoft Research Pex, a state-of-the-art tool based on dynamic symbolic execution, has been shipped as IntelliTest in Visual Studio 2015. For test inputs automatically generated by such tool, to supply test oracles (beyond just uncaught runtime exceptions or crashes), developers can write formal speci cations such as code contracts in the form of preconditions, postconditions, and class invariants. However, just like writing other types of formal speci cations, writing code contracts, especially postconditions, is challenging. In the past decade, parameterized unit testing has emerged as a promising alternative to specify program behaviors under test in the form of unit tests. Developers can write parameterized unit tests (PUTs), unit-test methods with parameters, in contrast to conventional unit tests, without parameters. PUTs have been popularly supported by various unit testing frameworks for .NET along with the recent JUnit framework. However, there exists no study to o er insights on how PUTs are written by developers in either proprietary or open source development practices, posing barriers for various stakeholders to bring PUTs to widely adopted practices in software industry. To ll this gap, in this paper, we present the rst empirical study of parameterized unit testing conducted on open source projects. We study hundreds of parameterized unit tests that open source developers wrote for these open source projects. Our study ndings provide valuable insights for various stakeholders such as current or prospective PUT writers (e.g., developers), PUT framework designers, test-generation tool vendors, testing researchers, and testing educators.