An interactive big data processing/visualization framework
Mackenzie Jorgensen, Jonathan Spohn, C. Bunn, Shi Dong, Xiangyu Li, David R Kaeli · 2017
Big data applications continue to grow in number and are challenging our computing capabilities. Given the rate of data collection, which grows exponentially, we need enhanced software solutions to analyze this mountain of data. As data visualization becomes critical in analytics, we want to provide engineers and scientists with the ability to explore their datasets interactively, providing state-of-the-art machine learning analysis in a timely fashion. In this paper, we present our NICE framework, which provides interactive visualization/analytics of large datasets. The framework has been developed leveraging the best practices in software engineering standards. Our framework is composed of three parts: a computing engine in C++ and CUDA as backend, a Python-to-C++ support end-to-end communication and zero-overhead data transfer, and a Python front-end developed for visualization. We leverage optimized software libraries for BLAS operations, Python-C++ interactions, and acceleration. We also developed several efficient machine learning algorithms to support interactive data analytics, targeting both multi-core CPUs and many-core GPUs.