Shepherd: Seamless Integration of Service Workflows into Task-Based Workflows through Log Monitoring
Md Saiful Islam, Douglas Thain · 2024
Traditional workflow managers focus on coordinating discrete tasks: actions that run to completion. However, emerging workflows require persistent services that must be managed alongside traditional tasks. We introduce Shepherd, a local workflow manager that runs services as a task, enabling them to be seamlessly integrated into larger distributed workflows. By inferring service states through log outputs and file creations, Shepherd enables the coordinated startup and shutdown of dependent services without modifying their original code. We demonstrate Shepherd’s effectiveness in large-scale drone simulations, where it enhances workflow flexibility, reliability, and comprehensive logging and visualization.