Towards Unified Data Ingestion and Transfer for the Computing Continuum
Muhammad Arslan Tariq, Ovidiu-Cristian Marcu, Grégoire Danoy, Pascal Bouvry · 2023
The computing continuum can enable new, novel big data use cases across the edge-cloud-supercomputer spectrum. Fast and high-volume data movement workflows rely on state-of-the-art architectures built on top of stream ingestion and file transfer open-source tools. Unfortunately, users struggle when faced with dealing with such diverse architectures: stream ingestion was designed for small-size datasets and low latency, while file transfer was designed for large-size datasets and high throughput. In this paper, we propose to unify ingestion and transfer, while introducing architectural design principles and discussing future implementation challenges.