A Optimization Algorithm for Association Rule Based on Spark Platform
Yongxiong Zhang, Liangming Wang · 2020
Compared with the limitations of hardware resources for the serial algorithm of association rule, the parallel algorithm has obvious advantages in large data processing and operation efficiency. This paper tries to design parallel Apriori algorithm and FP-Growth algorithm by the Spark platform. Through experiment contrast, FP-Growth parallel algorithm (PS-FP-Growth algorithm) is better. The PS-FP-Growth algorithm deals with unbalanced nodes by grouping them and FP Tree structure is complex. According to these problems, this paper presents the optimization and improvement of the balanced grouping strategy of node tasks and the non-frequent item merging pruning strategy. By the final analysis, the improved algorithm (PS-MP-FP-Growth) is not only more efficient, but also more stable.