A MapReduce based distributed LSI
Yang Liu, Maozhen Li, Suhel Hammoud, Nasullah Khalid Alham, Mahesh Ponraj · 2010 Seventh International Conference on Fuzzy Systems and Knowledge Discovery · 2010
Latent Semantic Indexing is a widely used text mining technology nowadays due its effectiveness in dealing with the problems of synonymy and polysemy within a proper matrix scale. However LSI is enormously computationally intensive especially for processing large scale data. And effective solution is to increase the computational power available to LSI using multiple computing nodes. In this paper we propose a novel MapReduce based distributed LSI using Hadoop distributed computing architecture to implement K-means algorithm to cluster the documents and then using LSI on the clustered results. We evaluated the performances of the proposed MapReduce based LSI and comparison are made with standalone LSI. The results show a great improvement of LSI's performance in terms of speed.