Large scale cluster analysis with Hadoop and Mahout
Felix Aronsson · Lund University Publications Student Papers (Lund University) · 2015
User generated data is getting more and more common. This data often expands in to hundreds of millions, if not billions, of data points. It is in the interest of every company with these vast amounts of data to make sense of them in one way or another. In machine learning, cluster analysis has been one way of trying to categorize data without supervision. Mahout is a library which runs on top of the Hadoop framework and tries to make cluster analysis (as well as other machine learning algorithms) arbitrarily scalable. This thesis focuses on using Mahout to cluster a large data set to see if the clustering algorithms in Mahout will scale to several millions of documents and tens of millions of dimensions. I find that while it is theoretically possible, there are several practical limitations that influence both the ability to run cluster analysis on such data sets, and also the results.