Dual direction big data download and analysis
Jameela Al‐Jaroodi, Nader Mohamed, Abdulla Eid · ACM SIGMETRICS Performance Evaluation Review · 2014
The term Big Data was recently coined as the amount of generated and stored digital data has grown so rapidly that it has become very hard to store, manage and analyze without coming up with new techniques that can cope with such challenges. Finding innovative approaches to support big data analysis has become a priority as both the research community and the industry are trying to make use of these huge amounts of available data. In this paper we introduce a new approach to enhance the overall big data analysis performance. The approach calls for utilizing data set replication, parallel download, and parallel processing over multiple compute nodes. The main concept calls for simultaneously parallelizing the download of the data (in partitions) from multiple replicated sites to multiple compute nodes that will also perform the analysis in parallel. Then the results are given to the client that requested the analysis.