Processing Load Prediction for Parallel FP-growth
Iko Pramudiono, Katsumi Takahashi · 2005
Load balancing is a dominant factor to achieve scalable parallel frequent pattern mining. In this paper, we examine some methods to predict processing load for parallel FP-growth algorithm. We propose item processing order based heuristic and load prediction function based on the path depth and other statistics which can be collected before the execution of mining process. We also propose sampling to predict statistics such as the number of iterations. Finally, we implement those methods to improve the initial distribution of processing units i.e. conditional pattern bases as well as the load balancing during the execution of those conditional pattern bases. The performance evaluation shows that sampling based load prediction and item ordering heuristics perform well for the initial distribution.