Document Clustering Through Non-Negative Matrix Factorization: A Case Study of Hadoop for Computational Time Reduction of Large Scale Documents
Bishnu Prasad Gautam, Dipesh Shrestha · 2010
Abstract — In this paper we discuss a new model for document clustering which has been adapted using non-negative matrix factorization method. The key idea is to cluster the documents after measuring the proximity of the documents with the extracted features. The extracted features are considered as the final cluster labels and clustering is done using cosine similarity which is equivalent to k-means with a single turn. This model was implemented using apache lucene project for indexing documents and mapreduce framework of apache hadoop project for parallel implementation of k-means algorithm. Since experiments were carried only in one cluster of Hadoop, the significant reduction in time was obtained by mapreduce implementation when clusters size exceeded 9 i.e. 40 documents averaging 1.5 kilobytes. Thus it is concluded that the feature extracted using NMF can be used to cluster documents considering them to be final cluster labels as in k-means, and for large scale documents, the parallel implementation using mapreduce can lead to reduction of computational time. We have termed this model as KNMF (K-means with NMF algorithm). Index Terms — Document Clustering, KNMF, MapReduce