Analysis of large-scale distributed knowledge sources via autonomous cooperative graph mining
Georgiy Levchuk, Andres Ortiz, Xifeng Yan · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 2014
In this paper, we present a model for processing distributed relational data across multiple autonomous heterogeneous computing resources in environments with limited control, resource failures, and communication bottlenecks. Our model exploits dependencies in the data to enable collaborative distributed querying in noisy data. The collaboration policy for computational resources is efficiently constructed from the belief propagation algorithm. To scale to large data sizes, we employ a combination of priority-based filtering, incremental processing, and communication compression techniques. Our solution achieved high accuracy of analysis results and orders of magnitude improvements in computation time compared to the centralized graph matching solution.