Type- and Workload-Aware Scheduling of Large-Scale Wide-Area Data Transfers
Rajkumar Kettimuthu · OhioLink ETD Center (Ohio Library and Information Network) · 2015
Data generated by experimental, simulation, and observational science is growing exponentially.The resulting datasets are often transported over wide-area networks for storage, analysis, or visualization.Analysis of wide-area transfer logs reveals that traffic is typically nonuniform.While the average load does not saturate the resources involved in wide-area data movement, bursty periods often saturate the system.Thus, efficient scheduling of data transfers can optimize the resource utilization and improve the overall performance of data transfers.Such scheduling requires the ability to control the bandwidth allocated to an individual transfer.Achieving this is difficult, however.End-to-end wide-area data transfers involve many components affecting overall transfer performance.These components include storage systems, host resources including the network interface cards, the local campus network at both the source and destination, and the wide-area network (WAN).Moreover, all these components are shared across multiple applications and multiple users.Depending on system configuration and the load, different resources can be bottlenecks at different instances.Also, the current state of the art in file transfer scheduling is best effort for all transfers.But other types of transfers exist, including near-real-time and flexible transfers.A data transfer scheduler can use this knowledge to effectively schedule different types of transfers.We investigate the file transfer scheduling problem, where transfers of different types among different endpoints must be scheduled in order to maximize pertinent metrics.extremely helpful, and a great friend.I have learned a lot from him over the years, especially about managing things, being organized, and setting expectations.vii I also thank Marc Snir, my division director at Argonne, for placing his trust in me and supporting me.I am grateful to Rich Carlson, program manager at DOE