Accelerating Data Movement Leveraging End-System and Network Parallelism
Jun Yi, Rajkumar Kettimuthu, Venkatram Vishwanath · 2012
Data volumes produced by simulation, experimental and observational science is rapidly increasing. This data needs to be moved from its source to another resource for analysis, visualization and archival pur- poses. The destination resource could be either local or remote. The data intensive science is critically de- pendent upon the high-performance parallel file and storage end systems to read/write and high-speed networks to move their enormous data between local and remote computing and storage facilities. 100 Gigabit per second networks such as DOE's Advanced Network Initiative (ANI), Internet2's 100G network represent a major step forward in wide area net- work performance. Effective utilization of these net- works requires substantial and pervasive parallelism, at the file system, end system, and network levels. Additional obstacles such as heterogeneity and time- varying conditions of network and end system arise that, if not adequately addressed, will render high performance storage and network systems extremely underperformed. In this paper, we propose a data movement system that dynamically and adaptively adjusts end systems and networks parallelisms in re- sponse to changing conditions of end systems and networks to sustain high-throughput for data trans- fers. We evaluate our system in multiple settings and show that (1) in a homogeneous configuration, the design can achieve better throughput for light and medium workload than GridFTP and achieve com- parable throughput for heavy workload, (2) and in a heterogeneous configuration, the design can achieve several factors higher throughput for all workloads than GridFTP.