Data-driven Software Feature Validation: Common challenges and how to mitigate them.
Peter Edvard Hildén · Työväentutkimus Vuosikirja · 2024
Background: Recent studies say that there is an unfulfilled potential of data-driven decision making. Only few studies exist on challenges that lead to the unfulfilled potential in adoption of data-driven practices. Additionally, there is a lack of studies on best practices and mitigation methods that address the challenges with adopting data-driven practices. Aims: With this study we want to gather an understanding of challenges teams face when adopting data-driven software feature validation. Knowing about common challenges can help teams avoid them and thereby implement data-driven approaches more efficiently with less risk. Additionally, this study aims to find effective mitigation methods to use when facing those challenges. By knowing effective mitigation strategies, teams can take actionable steps towards improving the quality of their data-driven validation efforts, leading to better software outcomes. Method: We asked 9 software practitioners in a semi-structured questionnaire about the challenges they face when adopting data-driven software feature validation and what mitigation methods they use to overcome those challenges. Results: We identified seven challenges that teams face when adopting data-driven software feature validation: Not knowing what data to collect, Not knowing how to use the data, Not getting enough data, Loving your own solution, Fixing the wrong thing, Not knowing how to collect data and Not prioritising software validation. Additionally, we found eight effective mitigation methods to address those challenges: Having an iterative process, Doing qualitative validation for insights, Raising the data expertise, Doing cross functional collaboration, Gathering reliable data, Having the big picture in mind, Defining clear metrics to measure success and Being transparent about data- driven validation. Conclusions: Teams striving to adopt data-driven software feature validation, will benefit from this study by being able to recognize common challenges and ultimately address them based on the suggested mitigation methods. By successfully mitigating challenges with data-driven adoption we will hopefully see the fulfilled potential of data-driven decision making soon. Our results show that it is important to start with Defining clear metrics to measure success as that will help with mitigating all the challenges found in this study.