A Novel Method of Adopting Graph Reduction for Resource Management in Parallel Computing Model
Chao Shen, Tong Weiqin · 2013
How to get the potential value from big data in real time has become a research focus. Now parallel computing model has been adopted the mainstream technology of mining big data by researchers, but the performances of model need to largely improve. In this paper, we build an isolated resource manager between parallel computing model and cluster to operate the resources. Furthermore, we use resource tree to show the resources state and adopt graph reduction of functional language to realize the dynamic and real time management of resources. Finally, based on the isolated resource manager, the resources of cluster such as CPU, memory, secondary storage, I/O and bandwidth could be released and applied instantly by the job, so the performances of parallel computing model has a greatly improvement.