Empowering Data Mesh with Federated Learning
Haoyuan Li, Salman Zubair TOOR · 2024
The evolution of data architecture has seen the rise of data lakes, aiming to solve the bottlenecks of data management and promote intelligent decision-making. However, this centralized architecture is limited by the proliferation of data sources and the growing demand for timely analysis and processing. A new data paradigm, Data Mesh, is proposed to overcome these challenges. In this decentralized architecture where data is locally preserved by each domain team, traditional centralized machine learning cannot conduct effective analysis across multiple domains, especially for security-sensitive organizations. To this end, we introduce a pioneering approach that incorporates Federated Learning into Data Mesh. This applied research article emphasizes the benefits of combining two distinct domains to achieve the best outcomes for industrial use cases.