Understanding System and Architecture for Big Data
Anne Gattiker, H Fadi, Ahmed Gheith, H. Peter Hofstee, Damir A. Jamsek, Jian Li, Evan W. Speight, Ju Wei Shi, Cheng Guan, Peter W. Wong · 2012
The use of Big Data underpins critical activities in all sectors of our society. Achieving the full transformative potential of Big Data in this increasingly digital world requires both new data analysis algorithms and a new class of systems to handle the dramatic data growth, the demand to integrate structured and unstructured data analytics, and the increasing computing needs of massive-scale analytics. In this paper, we discuss several Big Data research activities at IBM Research: (1) Big Data benchmarking and methodology; (2) workload optimized systems for Big Data; (3) case study of Big Data workloads on IBM Power systems. In (3), we show that preliminary infrastructure tuning results in sorting 1TB data in 14 minutes 1 on 10 Power 730 machines running IBM InfoSphere BigInsights. Further improvement is expected, among other factors, on the new IBM PowerLinux TM 7R2 systems.