Autoscaling MPI Cluster using OpenMPI and OpenStack cloud services
Silviu-Dumitru Pavăl, Cristian-Mihai Amarandei, Elena Şerban, Cătălin Mironeanu · 2023
In this paper, we present a novel architecture for an auto-scaling MPI cluster that leverages the capabilities of OpenMPI and OpenStack frameworks. Unlike traditional MPI clusters, which are typically static and expand to the maximum available resources, our proposed architecture aims to dynamically allocate resources in a cloud environment. This strategy is essential to mitigate high costs associated with static provisioning and ensure that the parallel tasks running on the cluster can fully benefit from the computational power of the cloud when needed. By integrating OpenMPI, a high-performance MPI implementation, with the flexible resource management capabilities of OpenStack, we enable the dynamic allocation and de-allocation of compute resources based on the cluster’s workload demands. This auto-scaling mechanism allows the cluster to scale up or down in response to changing computational requirements, optimizing resource utilization and reducing costs.