A loosely coupled metacomputer: co-operating job submissions across multiple supercomputing sites
Gregor von Laszewski · Concurrency Practice and Experience · 1999
This paper introduces a general metacomputing framework for submitting jobs across a variety of distributed computational resources. A first-come, first-served scheduling algorithm distributes the jobs across the computer resources. Because dependencies between jobs are expressed via a dataflow graph, the framework is more than just a uniform interface to the independently running queuing systems and interactive shells on each computer system. Using the dataflow approach extends the concept of sequential batch and interactive processing to running programs across multiple computers and computing sites in co-operation. We present results from a Grand Challenge case study showing that the turnaround time was dramatically reduced by having access to several supercomputers at runtime. The framework is applicable to other complex scientific problems that are coarse grained. Copyright © 1999 John Wiley & Sons, Ltd.