Implementing Low-Latency Machine Learning Pipelines Using Directed Acyclic Graphs
Abhishek Das, Nanda Kishore Gannamneni, Rakesh Jena, Raghav Agarwal, Prof. Sangeet Vashishtha, S. K. Jain · Journal of Quantum Science and Technology. · 2024
Low-latency machine learning (ML) pipelines are critical for applications that require real-time data processing and decision-making, such as fraud detection, autonomous driving, and financial trading. Achieving low latency in machine learning pipelines often involves complex data orchestration, model execution, and inference processes that need to be both efficient and scalable. One approach to optimize these processes is through the use of Directed Acyclic Graphs (DAGs). This research paper explores the design and implementation of low-latency ML pipelines using DAGs, highlighting their effectiveness in reducing computational overhead, managing dependencies, and ensuring efficient task execution. DAGs represent a flow of operations in a machine learning pipeline where each node corresponds to a task, such as data preprocessing, feature extraction, model training, or inference. The edges of the graph define the flow of data and the dependencies between these tasks. Using a DAG structure, tasks can be parallelized, synchronized, or executed asynchronously, which significantly reduces overall execution time and improves pipeline performance. By enabling the separation of independent tasks and optimizing execution order, DAGs can minimize latency compared to traditional linear or sequential execution models. The proposed framework leverages DAG-based data management and processing to address two key challenges in low-latency ML pipelines: efficient resource utilization and fault tolerance. Traditional pipelines often suffer from bottlenecks due to synchronous data transfer or redundant computations. With DAGs, tasks are only executed when their dependencies are met, and re-computation is avoided through effective caching mechanisms. Additionally, DAGs provide a modular architecture, allowing for easy experimentation, reconfiguration, and scaling to accommodate varying data volumes and model complexities. This research also introduces optimization techniques that enhance low-latency performance within DAG-based systems, such as minimizing inter-node communication overhead, using asynchronous task execution, and implementing distributed caching for intermediate results.