How do Secondary Studies in Software Engineering report Automated Searches?
Paramvir Singh, Matthias Galster, Karanpreet Singh · 2018
Context: Systematic literature reviews and mapping studies usually rely on automated searches of digital libraries to identify primary studies. Defining proper search strings, executing semantically similar searches on different libraries, and reporting limitations of searches increase the reliability of secondary studies. Objective: We aim to survey the current state of using automated searches in secondary software engineering studies. In particular, we aim at analyzing how automated searches are reported and at understanding the reproducibility of secondary studies. Method: We perform a preliminary tertiary study that covers 50 recently published representative secondary studies from different software engineering venues and subfields. Results: We found that most secondary studies complement an automated search with a manual search and use four or more digital libraries. Also, we found that the quality of reporting search strings is rather poor. Finally, we found that most secondary studies do not acknowledge limitations of automated searches and implications of limitations on study findings. Conclusions: Our findings highlight implications for researchers (e.g., to properly report the search process) and for reviewers (e.g., to execute search strings reported in papers). Also, our findings indicate that secondary studies are difficult to replicate.