Distributed parallel computing using windows desktop systems
David L. Hart, Douglas Grover, Matt Liggett, Richard Repasky, Corey Shields, Stephen C. Simms, Adam Sweeny, Peng Wang · 2004
Like many large institutions, Indiana University has thousands of desktop computers devoted primarily to running office productivity applications on the Windows operating system, tasks which are necessary but that do not use the computers’ full capacity. This is a resource worth pursuing. However, the individual desktop systems do not offer enough processing power for a long enough period of time to complete large scientific computing applications. Some form of distributed, parallel programming is required, to make them worth the chase. They must be instantly available to their primary users, so they are available only intermittently. This has been a serious stumbling block: currently available communications libraries for distributed computing do not support such a dynamic communications world well. This paper introduces Simple Message Broker Library (SMBL), which provides the flexibility needed to take advantage of such ephemeral resources. Condor [1] offers an approach to managing jobs on scattered computing resources that is well suited to this situation; there is a Windows version of Condor, although it does not at the time of this writing provide support for parallel computing. There are other systems for managing jobs in a distributed environment, such as Globus [2]. SMBL addresses a different problem: performing extended computations using a continually changing collection of small computers. We could not find a sufficiently fault-tolerant and wellbehaved PVM [3] implementation for Windows. MPI [4] implementations expect the same machines at the end of a job as at the beginning. This is only reasonable, since these libraries are generally used on dedicated systems. DOGMA [5] supports the desired type of computing, but only for applications written in Java. SETI@Home [6] does not provide a general-purpose framework. SMBL enables parallel computing on sporadically-available desktop systems by introducing a server to keep track of the processing nodes and route messages between them. The SMBL server acts as a communications broker for processes associated with a particular parallel job running on many different processors. SMBL is designed to work with heterogeneous systems. It is not a part of Condor, but they work well together. In conjunction, they can be used to run parallel jobs on Windows computers in an opportunistic fashion, without interfering with the computers’ primary users. Available as open source, SMBL is scalable, flexible and robust enough for a highly constrained and highly dynamic distributed computing environment, using ephemeral resources for massive computations.