Running scientific algorithms as array database operators: Bringing the processing power to the data
Simon Marcin, Andre Csillaghy · 2016
Array databases are well suited for storing and processing large multidimensional data. However, they usually run rather simple operations which only represent single steps of scientific algorithms. A way to run more complex logic is needed. In this paper, we study and test how to run entire scientific algorithms as native array database operators in a SciDB cluster. We present as use case our implementation of an iterative algorithm that reconstructs the distribution of plasma density in the solar corona at specific temperatures. This algorithm uses images series of the NASA spacecraft Solar Dynamic Observatory (SDO), which we stack in a 3-dimensional array. We measure for our use case a decrease of the overall runtime by an order of magnitude. We discuss different parameters used to scale up the array database cluster.