PipeMind: Toward a Multi-Agent Framework for Real-Time Feedback and Continuous Optimization in Analytics Pipelines
Hirad Rezaei, Fethi Rabhi, Amin Beheshti · 2025
Modern organizations depend on analytics pipelines for decision-making, yet integrating advanced ML/AI techniques often increases complexity and hinders optimal performance. This paper introduces PipeMind, a multi-agent framework that leverages large language models (LLMs) and specialized agents to provide real-time feedback and continuous optimization. Focusing on execution time, resource utilization, and throughput, PipeMind autonomously monitors pipeline performance, identifies bottlenecks, and recommends improvements. Its web-based infrastructure, built on RESTful APIs, seamlessly integrates with external data sources and cloud services. We demonstrate a working prototype with quantitative evaluations and a high-frequency trading case study, underscoring PipeMind's potential to enable fully autonomous analytics pipeline management.