Scalability Optimization in Cloud-Based AI Inference Services: Strategies for Real-Time Load Balancing and Automated Scaling
Yihong Jin, Ze Yang · 2025
With the fast growth of AI inference services on clouds, how to rapidly scale and ensure high performance under dynamic workloads has been demanding for a good solution. This study presents a novel framework for scalability optimization of cloud AI inference services, which aims at real-time load balancing and autoscaling strategies. Therefore, this model proposes a hybrid system, which consists of use of reinforcement learning to adaptively distribute load on servers and deep neural network to compute the demand for each user. By employing this multi-layered method, the system can predict workload variation patterns to preemptively allocate resources, which optimises resource utilisation and minimises latency. The key aspect is that this model not only provides the ability to recognise failure more effectively or improve overall fault tolerance, but with the use of decentralised decision making allows for a faster, more responsive scale up (and down) to assist with demand fluctuations. Experimental results show that the model improves load balancing efficiency by 35% and reduces response delay by 28%, so that compared to the traditional scaling solution, the optimization effect is obviously.