Multi-Objective Optimization Algorithm in Big Data Environment
Zhao Jie · 2023
None of the basic system software in big data domain can identify and use such algorithms. So the study evaluates hardware compression the performance of the method in big data environment is of great significance. The algorithm takes into account both the completion time of the optimization task and the load equilibrium value of the heterogeneous fog cluster, and adopts a nonlinear decreasing method to update the inertial weight value. The effectiveness of the proposed algorithm is verified by MATLAB simulation. The experimental results show that the algorithm improves the convergence rate at the same time as obtaining more search space at the early stage. Based on the KAEzip experiment, the performance of the classic Spark benchmarking program was evaluated in Spark. Tests of KAEzip show that: 1) Hardware compression algorithm can effectively improve Spark performance, such as KAEzip has up to 13.8% compression performance advantage over snappy, up to 7% decompression advantage and up to 5.7% performance potential in the actual application scenario; 2) The mismatch between the data transfer rate of the disk and the performance of the hardware compression algorithm is an important factor that restricts the performance of the hardware compression algorithm.