Scaling Approaches for Serverless Data Pipelines in Edge and Fog Computing Environments: A Performance Evaluation
Shivananda R. Poojara, Pelle Jakovits, Rajkumar Buyya, Satish Narayana Srirama · ACM Transactions on Autonomous and Adaptive Systems · 2025
The rise of Internet of Things (IoT) applications has led to massive data generation. However, dealing with such massive data is challenging. Nowadays, data pipelines are popular mechanisms used to properly deal with data operations at scale in the IoT continuum. Serverless Data Pipelines (SDP) is one such approach to performing event-driven data analysis on data streams. Data pipelines are composed of many components, and scaling the entire pipeline without leaving any bottlenecks is challenging. This study aims to assess the performance of scaling mechanisms in handling stochastic workloads efficiently and understanding critical resource utilization in fog environments. We applied workload-based techniques (Request per Second, Queue Length, Message Rate) and resource-based scaling (CPU) on SDP components of two IoT applications: Aeneas (long-running functions) and PuhatuMonitoring (short-running functions). Using Azure serverless workload patterns, we compared scaling approaches in real-time fog environments, evaluating QoS metrics like processing time and CPU utilization. Our analysis of suitability, using the weighted average scoring method on two QoS metrics, revealed that for compute-intensive tasks, the resource-based scaling approach works effectively for jump, steady, spike, and fluctuation workloads. For short execution time tasks, workload-based scaling suits all four workloads.