Research on Distributed Parallel Eclat Optimization Algorithm
Huang Qiufeng, Qiang Li, Huang Shiya, Chen Yingcong · 2020
Frequent Itemset Mining (FIM) is the core of tasks such as association rules and sequential pattern mining. With the increasing amount of data, traditional FIM algorithms become inefficient due to excessive resource requirements or high communication costs. In this paper, the Eclat algorithm in the frequent itemset mining algorithm is taken as the research point, and the parallel Eclat optimization algorithm BPEclat (Balanced Parallel Eclat) based on Spark is proposed to solve the performance shortcoming of Eclat algorithm in serial processing large-scale data. The algorithm is improved and optimized from many aspects: combining the pre-pruning and post-pruning depth pruning strategies to reduce the calculation of irrelevant itemset, compressing the candidate set size; using the prefix term to divide the data set, and using the range partitioning idea to balance the calculations node load, improve the parallel computing power of the algorithm. The experimental results show that the proposed BPEclat algorithm reduces the candidate set size by 25.3% and the time consumption by 32.5%. Therefore, it is possible to process massive amounts of data more efficiently and reliably, and has good scalability and universality.