Octopus: Decentralized Workflow-granular Scheduling for Serverless Workflow
Keming Wang, Liaoliao Feng, Ligang He, Chenlin Huang, Fengyuan Yu, Tao Xie · 2025
With the continuous development of Serverless Computing, Serverless applications composed of multiple finegrained functions have been widely applied in various fields of real life. As a pre-defined logical abstraction of Serverless applications, Serverless Workflow describes the dependencies and data flow between functions, and is the mainstream paradigm of modern Serverless Computing. However, our investigation shows that traditional Master-Worker-based, function-granular Serverless Workflow Management Systems are no longer suitable for the multi-function composition and unpredictable high concurrency characteristics of current Serverless Workflow. The seemingly insignificant scheduling overhead of Serverless Workflows has become a non-trivial factor affecting the execution efficiency and scalability of Serverless Workflows. Therefore, we proposed a workflow-granular management paradigm and decentralized control to address these challenges. Following these methodologies, we implement Octopus to enable efficient workflow scheduling and execution across different levels of concurrency and cluster scales. Experiments indicate that, in high-concurrency environments, Octopus achieves up to a 90× reduction in scheduling overhead and can enhance execution efficiency by 7.5×. As the cluster size increases, Octopus shows acceptable overhead and high scalability.