Using Traditional Data Analysis Algorithms to Detect Access Patterns for Massive Data Processing
Jiaqi Zhao, Jie Tao, Lizhe Wang, Rajiv Kumar Ranjan, Joanna Kołodziej · 2013
The data sets produced in our daily life is getting larger and larger. How to manage and analyze such big data is currently a grand challenge for scientists in various research fields. MapReduce is regarded as an appropriate programming model for processing such big data. However, the users or developers still need to efficiently program appropriate data processing actions related to their analytics requirements. In other words analytics actions in MapReduce is not portable across different big data types. In this paper we propose to adopt traditional data clustering algorithms to automatically analyze large data sets. We applied this approach to process performance data on distributed shared memory machines for detecting the application access patterns. The advantage is that application developers need not write codes to understand the runtime access behavior of their applications. We optimized several benchmark applications based on the analysis results and the experiments show a considerable improvement in terms of execution time and speedup.