Automatically Resolving Data Source Dependency Hell in Large Scale Data Science Projects
Laurent Boué, Pratap Kunireddy, Pavle Subotić · 2023
Dependency hell is a well-known pain point in the development of large software projects and machine learning (ML) code bases are not immune from it. In fact, ML applications suffer from an additional form of dependency hell, namely, data source dependency hell. This term refers to the central role played by data and its unique quirks that often lead to unexpected failures of ML models which cannot be explained by code changes. In this paper, we present an automated data source dependency mapping framework that allows MLOps engineers to monitor the whole dependency map of their models in a fast paced engineering environment and thus mitigate ahead of time the consequences of any data source changes. Our system is based on a unified and generic approach, employing techniques from static analysis, from which data sources can be identified on a wide range of source artifacts. Our framework is currently deployed within Microsoft and used by Microsoft MLOps engineers in production.