Distributed Bayesian Network Learning Algorithm using Storm Topology
Fei Ding, Yi Zhuang · International Journal of Grid and Distributed Computing · 2018
The use of cloud computing enables access to high-capacity computing resources, which exploits parallel resources using parallel-machine learning algorithms such as Bayesian network structural learning algorithms.In this paper, we propose the distributed Bayesian network learning (DBNL) algorithm other than the MapReducebased methods based on the Storm topology framework.By using the calculation for one DAG or CPDAG as the basic computation unit, we divide the searching space to the minimum scale to store B-spaces or E-spaces in parallel.In order to detect loops in the search space, we design the mechanics that ensures a state being sent to the same computing node if it has already been previously scored.We also use the hash code of each state as the key of a tuple in the grouping stage to assign processing tasks efficiently.We present the DBNL algorithm in detail and demonstrate its flexibility by giving four adapted variants based on hill-climbing, K2, TPDA and GES algorithms.We perform experiments that approximate the linear speedup.We conduct experiments using four real-life datasets to test the performance of DBNL in setups with different data size, numbers of processing instances and scoring criterion.The experimental results obtained indicate that the percentage of parallel parts is up to 93.28%, and the performance improvements are more significant with larger datasets.