TeraGrid's Tools for Massive Data Movement.
Anthony Vu, Martin Margo, Patricia A. Kovatch, Christopher Jordan, Richard Moore, William E. Allcock · 2006
Abstract- As scientists generate multi-Terabyte sized data sets from geographically distributed high performance compute, visualization and other datagenerating resources, the need to share and transfer data efficiently between these resources becomes more important. On the TeraGrid [3], scientists use multiple, unique resources at each partner site and need the same files available. To help the scientists achieve their goal, we identified two standards-based approaches: GPFS-WAN [1] and GridFTP [2]. The General Purpose File System (GPFS) exported over a Wide Area Network (WAN) is a high performance parallel file system located in San Diego but available across the TeraGrid. GridFTP offers parallel data transfer services including a bulk file transfer facility. With custom tools, tuning and scripting enhancements, we offer these easy-to-use services to help scientists be productive.