Using Open Source Desktop Grids in Scientific Computing and Visualization
Zoran Constantinescu, Monica Vlădoiu · InTech eBooks · 2011
Scientific Computing is the collection of tools, techniques, and theories required to develop and solve on a computer, mathematical models of problems in science and engineering, and its main goal is to gain insight of such problems (Heat, 2002;Steeb et al., 2004; Hamming, 1987).Generally, it is difficult to understand or communicate information from complex or large datasets generated by scientific computing methods and techniques (computational simulations, complex experiments, observational instruments etc.).Therefore, support of Scientific Visualization is needed, to provide techniques, algorithms, and software tools that are necessary to extract and display properly important information from numerical data.Complex computational and visualization algorithms normally require large amounts of computational power.The computing power of a single desktop computer is not sufficient for running such complex algorithms, and, traditionally, large parallel supercomputers or dedicated clusters were used for this job.However, very high initial investments and maintenance costs limit the large-scale availability of such systems.A more convenient solution, which is becoming more and more popular, is based on the use of non-dedicated desktop PCs in a desktop grid computing environment.Harnessing idle CPU cycles, storage space and other resources of networked computers, to work together, on a particularly computational intensive application, perform this job.Increasing power and communication bandwidth of desktop computers provides for this solution as well.In a Desktop Grid (DG) system, the execution of an application is orchestrated by a central scheduler node, which distributes the tasks amongst the worker nodes and awaits workers' results.It is important to note that an application only finishes when all tasks have been completed.The attractiveness of exploiting desktop grid systems is further reinforced by the fact that costs are highly distributed: every volunteer supports her resources (hardware, power costs and Internet connections), while the benefited entity provides management infrastructures, namely network bandwidth, servers and management services, receiving in exchange a massive and otherwise unaffordable computing power.The typical and most appropriate application for desktop grid comprises independent tasks (no communication exists amongst tasks) with a high computation to communication ratio (Domingues, Silva & Silva, 2006;Constantinescu, 2008).The usefulness of desktop grid computing is not limited to major high throughput public computing projects.Many institutions, ranging from www.intechopen.comAdvances in Grid Computing 148 academics to enterprises, hold vast number of desktop machines and could benefit from exploiting the idle cycles of their local machines.In fact, several studies confirm that CPU idleness in desktop machines averages 95% (Domingues, Silva & Silva, 2006;Constantinescu, 2008;Vladoiu & Constantinescu, 2009b).In the work presented in this chapter, the central idea has been to build a desktop grid computing framework and to prove its viability by testing it in some scientific computing and visualization experiments.We present here QADPZ, an open source system for desktop grid computing, which enables users from a local network or even Internet to share their resources.It is a multi-platform, heterogeneous system, where different computing resources from inside an organization can be used.It can also be used for volunteer computing, where the communication infrastructure is the Internet.QADPZ supports the following native operating systems: Linux, Windows, MacOS and Unix variants.The reason behind the native support for multiple operating systems, and not only for one (Unix or Windows, as other systems do), is that, often in real life, this kind of limitation restricts very much the usability of dekstop grid computing.QADPZ provides a flexible object-oriented software framework that makes it easy for programmers to write various applications, and for researchers to address issues such as adaptive parallelism, fault-tolerance, and scalability.The framework supports also the execution of legacy applications, which for different reasons could not be rewritten, and that makes it also suitable for other domains as business.It also supports either low-level programming languages as C and C++ or high-level language applications, like for example Lisp, Python, and Java, providing the necessary mechanisms to use such applications in a computation.Consequently, users with various backgrounds can benefit from using QADPZ.The flexible, object oriented structure and the modularity of the system allows improvements and further extensions to other programming languages to be made easily.We have developed a general-purpose runtime and an API to support new kind of high performance computing applications, and therefore to benefit from the advantages offered by desktop grid computing.We show how distributed computing grid extends beyond the master-worker paradigm, typical for such systems, and provide QADPZ with an extended API which supports in addition lightweight tasks creation and parallel computing, using the Message Passing Interface paradigm (MPI).The C/C++ programming language is directly supported by the API.QADPZ supports parallel programs running on the desktop grid, by providing and API in the C/C++ language, which implements a subset of the MPI standard.This extends the range of applications that can be used in the system to already existing MPI based applications, like for example parallel numerical solvers, from computational science, or parallel visualization algorithms.Another restriction of existing systems, especially middleware based, is that each resource provider needs to install a runtime module with administrator priviledges.This poses some issues regarding data integrity and accessibility on providers computers.The QADPZ system tries to overcome this by allowing the middleware module to run as a non-privileged user, even with restricted access, to the local system (Constantinescu, 2008; Vladoiu & Constantinescu, 2008a).QADPZ provides also low-level optimizations, such as on-the-fly compression and encryption for communication.The user can choose from different algorithms, depending on the application, improving both the communication overhead imposed by large data transfers and keeping privacy of the data.The system goes further, by providing an experimental, adaptive compression algorithm, which can transparently choose different www.intechopen.comUsing Open Source Desktop Grids in Scientific Computing and Visualization 149 algorithms to improve the application.QADPZ support two different protocols (UDP and TCP/IP) in order to improve the efficiency of communication (Constantinescu, 2008; Constantinescu & Vladoiu, 2009a).Free availability of the source code allows its flexible installations and modifications based on the individual needs of research projects and institutions.In addition to being a very powerful tool for computationally-intensive research, the open-source availability makes QADPZ a flexible educational platform for numerous small-size student projects in the areas of operating systems, distributed systems, mobile agents, parallel algorithms, and others.Moreover, free or open source software constitutes a natural choice for modern research, as well, because it encourages integration, cooperation and boosting of new ideas, in a very effective way (Cassens & Constantinescu, 2003;Constantinescu, 2008).QADPZ has been built in top of an extended master-worker conceptual model.The extensions and refinements to the original master-worker paradigm concern mainly both the reduction of the time consumed for delays in communications and the increase of the period of time in which the workers perform computations.This goal is achieved by making use of different methods and techniques.The most important of them are as follows: pulling or pushing of work units, pipelining of the work-units at the worker, sending more work-units at a time, adaptive number of workers, adaptive timeout interval for work units, multithreading, redundant computation of the last work-units to decrease the time to finish, overlapped communication and computation at both the workers and the master, and use of compression to trim down the dimension of messages (Constantinescu 2008;Vladoiu & Constantinescu 2008b).Our work has also shown that that the use of desktop grid computing should not be limited to only master-worker type of application, but it can be used also for more fine-grained parallel applications, in the field of scientific computing and visualization, by performing some experiments in those domains.Thus, we have used QADPZ in several experiments: geophysical circulation modelling, fluid flow around a cylinder, both simulation and visualisation and so on (Constantinescu, 2008; Constantinescu & Vladoiu, 2009a).It is worth to mention that to the present QADPZ has over a thousand four hundred users who have download it, and that many of them use it for their daily tasks.They constantly give feedback on the system, ask for support in using it for their research and development tasks, or discuss about it in specialized discussion forums.QADPZ is also actively used in universities where students get work assignments based on its features (Constantinescu, 2008;QADPZ, 2010).The chapter structure is as follows: the second section reveals other major desktop grid environments, such as BOINC, distributed.net,Condor and XtremWeb, along with examples of both desktop grids generated with their support and projects that use these environments.The third section describes briefly the QADPZ system.It starts with a justification for our endeavour to build a new DG system, and continues with the main capabilities of QADPZ, and with the improvements of the master-worker model implemented in the system.Within the next section, some scientific and visualization experiments performed with QADPZ's support are presented: geophysical circulation modelling within the Trondheim fjord, fluid flow around a cylinder -simulation and fluid flow around a cylinder -visualization.In the last section we will present some conclusions and some future work ideas that aim to improve both the conceptual model and the QADPZ system.We also invite the interested enthusiastic developers in the open-source community to join our development team and we appreciate any kind of feedback.