Deterministic annealing and robust scalable data mining for the data deluge
Geoffrey Fox · 2011
We describe data analytics on large systems using a suite of robust parallel algorithms running on both clouds and HPC systems. We apply this to cases where the data is defined in a vector space and when only pairwise distances between points are defined. We introduce new O(N logN) algorithms for pairwise cases, where direct algorithms are O(N2) for N points. We show the value of visualization using dimension reduction for steering complex analytics and illustrate the value of deterministic annealing for relatively fast robust algorithms. We apply methods to metagenomics applications.