Routing and Scheduling Large File Transfers over Lambda Grids
Amitabha Banerjee, Wu-chun Feng, Biswanath Mukherjee, Dipak Ghosal · 2005
Abstract — In many application domains, there exists a need to aggregate information from information repositories distributed around the world. In an effort to better link these resources in a unified manner, middleware researchers put forth the notion of a grid. With this context, we consider the problem of aggregating files from distributed databases to a (grid) computing node over a lambda grid. The challenge is to identify concurrent routes (i.e., circuit-switched paths) in the lambda grid network, along which files should be transmitted, and to schedule the transfers of these files over their respective circuits. To address this challenge, we propose a hybrid approach that combines offline and online scheduling. The approach first constructs an offline schedule based on past profiling of transfer rates. Then, as files are being transferred, the schedule is modified online, depending on the amount of time that it actually took to transfer the files. The objective is to minimize the total time required for data aggregation. To demonstrate the effectiveness of our approach, we present experimental results using grid nodes running over an emulated lambda grid topology. I.