Open Source Exploratory Analysis of Big Earth Data With NEXUS
Thomas S. Huang, Edward Armstrong, N. T. Chung, Eamon Ford, Frank R. Greguska, Joseph C. Jacob, Brian D. Wilson, Elizabeth Yam, Alice Yepremyan · Special publications · 2022
Almost all of the existing Earth science data analysis solutions are built to work directly on large archives of files. Since much of the Earth science data is stored in files and organized by time, these solutions spent significant amounts of their time in opening and subsetting files stored on common file systems. As the size of the data required for the analysis increases, these traditional analysis solutions get slower and slower. NEXUS is a data-intensive analysis solution, part of the Apache Science Data Analytics Platform (SDAP) ( https://sdap.apache.org ), developed with a new approach for handling science data to take advantage of the parallel analytics architectural approach and the elasticity of the modern clustering and cloud computing environments. Most traditional Earth science analysis solutions are designed for vertical scaling, that is, buy a bigger and faster (sometime specialized) computer in order to handle more data and to improve execution speed. NEXUS is designed for horizontal scaling by partitioning geospatial array data into small tiles and distributing the computing to all the network-connected computing nodes. Using a horizontal-scale computing model, such as MapReduce, NEXUS is able to achieve hundreds of time speedups over traditional analysis solutions. This chapter will discuss the design of NEXUS along with some benchmarking comparisons and the applications of this open source technology.