Design of a framework for serverless distributed data processing using queues
Oleksandr O. Kyrychenkо, С. Е. Остапов, Оксана Леонідівна Кириченко · Eastern-European Journal of Enterprise Technologies · 2025
This study's object is the organization of distributed data processing in cloud environments using serverless computing. The evolution of serverless computing has led to a change in approaches to building the architecture of applications deployed in the cloud. The ability to abstract from infrastructure management while achieving cost-effectiveness is becoming a significant task requiring new tools. One such tool is a new framework that makes it possible to scale computing resources depending on the workload dynamically, store the state of the computing process using DynamoDB, and provide real-time progress tracking. A serverless application for the distributed generation of PDF documents has been developed and deployed to test the proposed framework. The real load was emulated using Locust; files containing 1,025,132 records were fed to the application input. The results of the experiments showed that the application started to work 25.8% faster, the throughput increased by 21.3%, and the number of cold starts decreased by 3% compared to conventional scaling. Additionally, the main areas of further research on developing and improving the designed framework have been identified. This study provides possibilities for predictive automatic scaling using semi-Markov process models in a serverless environment. Unlike traditional reactive approaches, the proposed approach predicts changes in advance and proactively scales parts of the application, which makes it possible to reduce delays and avoid cold starts. The framework could be used to develop serverless applications for distributed data processing using message queues and the ability to monitor the processing in real time