Data extraction for systematic mapping study using a large language model - a proof-of-concept study in software engineering
Katia Romero Felizardo, Igor Steinmacher, Marcia Heloísa Tavares de Figueredo Lima, Anderson Deizepe, Tayana Uchôa Conte, Monalessa Perini Barcellos · 2024
Context: Systematic mapping studies (SMS) are adopted in Software Engineering (SE) to select and synthesize relevant literature on a research topic and, thus, support evidence-based decision-making. Performing SMS is effort-demanding and time-consuming. Hence, using tools is beneficial. Large Language Models (LLMs) such as ChatGPT–4.o can potentially accelerate repetitive activities, such as data extraction in SMS, saving time and effort. Goal: We conducted this work to evaluate and provide preliminary evidence on how ChatGPT–4.o can support data extraction in SMS. Method: We performed a proof-of-concept study and assessed the results’ accuracy of using ChatGPT 4.0 to extract data in one SMS compared to the results produced manually. Results: The accuracy of ChatGPT–4.o was 87.83%. Conclusions: Our preliminary findings suggest that entirely replacing the manual data extraction with ChatGPT–4.o is not recommended. However, employing ChatGPT for semi-automated data extraction to aid in evidence synthesis in SMS is promising.