Scaled and Projected Spectral Clustering with Vector Quantization for Handling Big Data
Vishal Nemade, Aditya Shastri, Kapil Ahuja, Aruna Tiwari · 2018
In this modern era, the advent of web technologies and social networking websites is generating a significant amount of data every day. In this scenario, where the data size is now reaching zetta bytes (i.e., 1021), its analysis is very important.Since spectral-based clustering algorithms provide more accurate results than traditional clustering algorithms, we focus on these algorithms. In our work, we propose a modified version of spectral clustering, which we call Projected Spectral Clustering (PSC). As the complexity of the PSC algorithm is Opn3q, where n is the size of the data, we use two variants of vector quantization sampling namely k-Means (KM) and Bisecting k-Means (BKM). To make our algorithm scalable for handling Big Data, we implement it on Apache Spark using two approaches for computing the Gaussian Kernel matrix, which is the most important step here (i.e. Map Reduce and Map Only). We call this algorithm Scalable PSC (SPSC).We measure the accuracy of SPSC using three evaluation criteria tested on a variety of different datasets. Our new algorithm gives good clustering accuracies. Further, we perform another set of experiments on a different number of cores to demonstrate runtime/ scalability efficiency of our algorithm. Finally, we prove this scalability by doing a complexity analysis.