Scaling Big Earth Science Data Systems Via Cloud Computing
Hook Hua, G. Manipon, Sujen Shah · Special publications · 2022
Observations of the Earth from remote sensing satellites are downloaded to various data systems on the Earth for processing, storage, and analysis. But with increasing complexity of the instruments, computer systems, science requirements, and algorithms, the amount of data to process has become so large that traditional science data systems and how data are handled can no longer be used. These forcing functions now require novel ways to handle the increased data such as moving all end-to-end data systems to be collocated in common cloud computing regions. The science data systems must also be rearchitected to be cloud enabled to be able to support petabyte-scale data processing and elastic processing demands and to utilize cloud-native services to be able to scale up efficiently to support the new data processing requirements. Science Data Systems (SDSes) provide the capability to develop, test, process, and analyze instrument observational data efficiently, systematically, and at large scales. SDSes ingest the raw satellite instrument observations and process them from low-level instrument values into higher level observational measurement values that compose the science data products. Before the advent of the cloud paradigm, the machinery for the SDS had mostly been hosted and operated on-premise at data centers owned and operated by the respective stakeholders. However, technological advancements in the satellite platform, resource availability, instrument sensitivities, communications downlink bandwidth, and improved science processing algorithms have resulted in increasingly larger mission science data volumes and data rates. These larger requirements and cost constraints are forcing functions for moving science data systems as well as other associated data systems in the enterprise onto the cloud. However, at big data scales, anomalous system behaviors such as resiliency and stability of the data system must also be addressed.