Enabling large-scale dataset analysis in resource-constrained environments through application-aware preprocessing
Jack Parry · Queensland University of Technology · 2020
In many computing applications, such as system monitoring, fault diagnostics, and bioinformatics, large datasets must be analysed. As the volume of such datasets increases inexorably, industry-standard analysis tools struggle to produce meaningful results in a reasonable amount of time or may fail to work at all. More efficient analysis software may not exist, and higher-performance computing environments may be prohibitively expensive or unsuitable for use in the field. We develop and present the technique of Application-Aware Preprocessing, incorporating user requirements directly into the data analysis process. The technique proves successful enough to allow large-scale dataset analysis in resource-constrained environments.