Unlocking the compute continuum: Scaling out from cloud to HPC and HTC resources
D. Ciangottini, D. Spiga, Shiraz Memon, Andrea Manzi, Andrej Filipcic, A. Troja, F. Fanzago, Giulio Bianchini, Massimo SGARAVATTO, Teo Prica, T. Boccali, T. Tedeschi · EPJ Web of Conferences · 2025
In a geo-distributed computing infrastructure with heterogeneous resources (HPC and HTC and possibly cloud), a key to unlock an efficient and user-friendly access to the resources is being able to offload each specific task to the best suited location. One of the most critical problems involves the logistics of wide-area, multi-stage workflows that move back and forth between multiple resource providers. We envision a model where such a challenge can be addressed enabling a “transparent offloading” of containerized payloads using the Kubernetes API primitives creating a common cloud-native interface to access any number of external hardware machines and type of backends. Thus we created the interLink project, an open source extension to the concept of Virtual-Kubelet with a design that aims for a common abstraction over heterogeneous and distributed backends. interLink is developed by INFN in the context of interTwin, an EU funded project that aims to build a digital-twin platform (Digital Twin Engine) for sciences, and the ICSC National Research Center for High Performance Computing, Big Data and Quantum Computing in Italy. In this talk we first provide a comprehensive overview of the key features and the technical implementation. We showcase our major case studies, such as the scale-out of an analysis facility, and the distribution of ML training processes. We focus on the impacts of being able to seamlessly exploit world-class EuroHPC supercomputers with such a technology.