An Evaluation of the Impact of Code Generation Tools on Software Development
Luiz Fernando Mendes Osório, Pedro de Alcântara dos Santos Neto, Guilherme Amaral Avelino, Werney Ayala Luz Lira · 2025
Context: The rise of AI-assisted tools like GitHub Copilot aims to improve productivity in software development, raising questions about their practical impact on developer performance and code quality. Problem: While AI-assisted tools are promoted for enhancing productivity, empirical evidence on their real-world impact remains limited. This study addresses whether such tools improve programming workflows, focusing on developers with varying experience levels and examining effects on coding quality and efficiency. Solution: An empirical study is conducted to assess Copilot’s effectiveness through experiments with student developers, focusing on task efficiency and code correctness, guided by established indicators identified in the literature. IS Theory: Grounded in the Task-Technology Fit (TTF) theory, this research explores the extent to which Copilot aligns with programming tasks and meets user needs, ensuring its relevance to practical development scenarios. Method: A literature review identified key indicators for evaluating AI-assisted code generation, focusing on task completion time and code correctness. Data collection involved task-based experiments with students, followed by quantitative analysis comparing performance with and without Copilot. Summary of Results: Findings indicate that Copilot can significantly reduce task completion time. However, no statistically significant differences were observed in code correctness, suggesting that while Copilot improves efficiency, its recommendations require careful review to ensure quality. Contributions and Impact on IS Field: This study contributes empirical insights into AI-assisted coding’s advantages and challenges. The results inform academia and industry about best practices for implementing such tools effectively, emphasizing their potential for accelerating development while maintaining the need for human oversight.