Parallel Frequent Itemset Mining Algorithm and Optimization Based on Spark
Chunran Liu · 2023
Frequent item set mining is an important task in data mining, which can help us find the variable combinations that occur frequently in the data set. In order to solve the problem of frequent itemset mining in large-scale high-dimensional data, we propose a parallel frequent itemset mining algorithm based on Spark, OSFPG. The algorithm is optimized on the basis of the traditional FP-growth algorithm, and the number of candidate item sets is reduced by constructing a special data structure called FP tree, thus improving the efficiency of frequent item set mining. The algorithm mainly uses the idea of load balancing to optimize the grouping strategy. It also considers two factors of partition computation and FP-Tree size to ensure that the total load between each group is approximately equal. On the basis of Spark based SFPG algorithm, we implement this optimized algorithm. Based on the SFPG algorithm, it is improved to dynamically allocate computing resources and carry out distributed computing, so that the intermediate results generated in the operation process are saved in memory, thus effectively reducing the data consumption. In order to verify the performance of OSFPG algorithm, we apply it to a real data set for experiments. Experimental results show that compared with the traditional FP-Growth algorithm, the proposed algorithm has higher efficiency and lower time complexity when processing large-scale high-dimensional data. In addition, the algorithm can deal with dynamically updated data effectively and has strong practicability.