Semi-Automatic Hybrid Software Deployment Workflow in a Research Computing Center
Fang Cherry Liu, Ronald O. Rahaman, Michael D. Weiner, Eric Coulter, Deepa Phanish, Jeffrey N. Valdez, Semir Sarajlic, Rubén Lara, Pam Buffington · Practice and Experience in Advanced Research Computing · 2023
The software stack is an essential component in any High Performance Computing (HPC) cluster, and it is critical to optimize the usage of the underlying computing resources while simultaneously providing the best user experience. Georgia Institute of Technology (GT) Partnership for an Advanced Computing Environment (PACE) maintains a heterogeneous system with around 2000 compute nodes across five different clusters on two different schedulers (Torque and Slurm) and two different parallel filesystems (GPFS and Lustre). This diversity calls for multiple software stacks that pose a significant challenge to delivering software support efficiently. Hardware upgrades, filesystem changes, and scheduler replacements all lead to software stack changes ranging from updating the MPI libraries to rebuilding the whole stack. In the past five years, PACE has rebuilt the software stack four times due to these events. To reduce the build time, Spack, GitLab, and ReFrame tools were adopted. Spack, an automatic software building tool, allows us to set up more than half of the software stack quickly; GitLab provides git tags and issues to handle Dev-Test-Prod workflow coordination within the team; and ReFrame provides a systematic way to test the whole software stack automatically. By integrating these three tools alongside manual software installations, we have created a semi-automatic hybrid software deployment workflow that is presented in this paper. We share our experience to contribute to the body of work and useful tradition of HPC centers sharing their practices with the community.