The Development of Real-time Large Data Processing Platform Based On Reactive Micro-Service Architecture
Haidong Lv, Tianzhi Zhang, Zhenhan Zhao, Jiahui Xu, Tianyu He · 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC) · 2020
The big data technologies has been used in all the field of industry, and now the automatic controller system also has the requirement to use big data techniques to analysis the read time big data to support the business need. This paper provides the solution for real time big data processing in industry controller system by combination of Spring Cloud Stream micro service, Kafka and Apache Spark. The paper also discusses how to develop event driven system with stream API of Spark and Spring for Apache Kafka Stream for real time big data processing and analysis. The construction of real time big data platform infrastructure and configuration in this platform has also been discussed.