Tracelab: Reproducing Empirical Software Engineering Research
Evan Moritz · W&M Publish (College of William & Mary) · 2013
The ability to reproduce experiments in software engineering research is a hidden issue in validating and improving on new approaches.The lack of tool support, data availability, implementation-specific details, and even minute environment differences all contribute to problems for researchers attempting to investigate new ideas.In this thesis, I present examples of how unpublished details of an approach can drastically change the results.To address this issue, I promote the use of TraceLab, a research instrument designed to perform and share software engineering experiments in their entirety with accurate results.I leverage TraceLab's ability to extend its framework with new tools and functionality to create a new Component Library (CL) and Component Development Kit (CDK) designed to provide researchers with all of the tools necessary to evaluate and improve new techniques.To discover which tools to include, I perform a systematic mapping study of publications from a subset of top international software engineering conferences in the past 10 years.Based on these results, I implemented the most popular tools and techniques in the CL and CDK.I show that by using the CL and CDK in TraceLab, 37% of the approaches identified in the mapping study can be completely recreated, with an additional 37% of approaches missing only 1 technique.Lastly, I reproduce examples of existing software engineering approaches that provides a working body of knowledge in order to drive new research.