Study of Data Transfer Nodes Infrastructure in Enabling Big Data Movement between Research and Cloud Storage Networks
Damir Regvart, Miljenko Mikuc, Vedran Dakić · 2023
Data research projects generate huge amounts of data, which are usually stored in cloud networks for processing and storage. Transferring large amounts of data between multiple research sites, often within protected scientific DMZs (Demilitarized Zone) and cloud storage nodes, requires high-speed network connections.To ensure efficient data transfer and processing, parallel data streams are used between multiple research sites, with dedicated cloud infrastructure for processing and storage of a particular experiment. Since multiple nodes or sites are involved in data transfer, DTNs (Data Transfer Nodes) are used to solve the problems of fast data flow and performance between nodes that are not related to network throughput. DTNs include software transfer tools and high-end hardware components that are tasked with preparing data for transfer.Several research groups are working to establish multiple DTN instances as part of a large-scale scientific data network. This paper provides a brief overview of the current state of the art in the use of DTN infrastructure and the challenges associated with preparing data for transfer.