A Beowulf-Style Cluster Processing Concept for Large Volume Imaging Spectroscopy Data
Jason Brazile, Michael E. Schaepman, Daniel R. Schlapfer · 2003
The commodity availability of inexpensive computers, large capacity hard drives, and local area network infrastructure has increased the viability of using cluster computing (e.g. Beowulf-style Linux clusters) for even small to medium data processing projects. As a consequence, it is becoming increasingly prominent in numerous earth observing applications [1]. However, there are many factors to be considered affecting how such a processing cluster can be built and used efficiently. Foremost, a fundamental understanding of the interplay between computation and I/O activities of the application is required [2]. It is also important to be familiar with available state-of-theart software components which can be used for the wellunderstood portions of application processing (e.g. FFTW [3] for convolutions). And when the application requires novel algorithms, an understanding of modern computer architectural concepts such as superscalar pipelining and cache management can make a significant difference in processing efficiency [4]. In addition to technical requirements, other factors such as the inhomogeneity of developer skills and resources, and deployment and future maintenance /upgrade expectations should be addressed. Here, we address each of these factors and discuss how they were analyzed to model a clustering solution for the calibration and processing of large volume imaging spectroscopy data for the European Space Agency's Airborne Prism Experiment (APEX).