Apache spark based distributed self-organizing map algorithm for sensor data analysis
Madhura Jayaratne, Damminda Alahakoon, Daswin De Silva, Xinghuo Yu · IECON 2017 - 43rd Annual Conference of the IEEE Industrial Electronics Society · 2017
The proliferation of Internets of Things (IoT) technologies in both industrial and non-industrial settings has led to the accumulation of Big Data sets. Analysis of these high-volume, high-velocity datasets require advanced processing techniques that incorporate parallel and distributed computations. In this paper, we present a novel distributed self-adaptive neural-network algorithm, the Distributed Growing Self-Organizing Map (DGSOM) algorithm to address the growing need for unsupervised machine learning of Big Data sets on distributed computing environments. The algorithm was tested on a Big Data set of sensor recordings of human activity collected from wearable devices, 2.8 million records. Results indicate that the distributed algorithm significantly reduces execution time compared to its serial counterpart. Moreover, the self-adaptive nature and controlled growth of the algorithm demonstrates data-driven structure adaptation and multi-granular pattern analysis. Overall, the proposed algorithm addresses the need for pattern discovery and visualization from Big Data sets generated by IoT devices which are increasingly commonplace in industrial scenarios.