Optimization of Flink distributed system with FPGA-based architecture
Juan Zhang, Jia You, Hao Yang, XiSheng Li, Yin Liu · 2024
With the continuous increase in the volume of Internet of Things (IoT) data, the limited number of servers deployed at the edge poses a pressing challenge for efficiently processing large-scale data streams. This study aims to explore the application of FPGA hardware acceleration technology to enhance the performance of the Apache Flink framework to meet the growing demands of users. The paper advocates the utilization of FPGA as a hardware accelerator for enhancing the performance of the Flink framework. This is achieved by harnessing the power of PCIe technology and integrating seamlessly with the OpenCL standard, which is tailored for heterogeneous systems. The JVM-FPGA communication mechanism is optimized using a data transmission pipeline mechanism. Through experiments and evaluations of two typical computationally intensive tasks, matrix multiplication and vector addition, it is demonstrated that the performance of the new framework is reduced in terms of latency, throughput is increased by 2.77 times, and CPU utilization is significantly enhanced. This study provides an innovative solution for efficiently processing largescale data streams in edge environments, successfully demonstrating that hardware acceleration can significantly improve the performance of the Apache Flink framework under resource constraints in edge environments, better meeting the continuously growing demands of IoT data.