Towards Asynchronous Distributed MCMC Inference for Large Graphical Models
Sameer Kumar Singh, Andrew McCallum · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · 2011
With increasingly cheap availability of computational resources such as storage and bandwidth, access to large amount of data has become commonplace. To perform inference over these millions of variables, there is a need to distribute the inference; however the dense, loopy structure with long-range dependencies makes the problem non-trivial. There has been some recent work in distributed inference for graphical models; however they make strong synchronization assumptions that we do not desire in large-scale models. In this work, we explore a number of approaches for distributed MCMC inference for graphical models in an asynchronous manner. The overall architecture consists of inference workers that independently access a data repository to obtain a set of variables and their current values. These workers then perform inference and write back the results to the repository. Although this approach provides more flexibility to the workers, without any “variable ownership” it leads to conflicts over values of a variable. We introduce a number of possible resolution strategies, and provide preliminary experiments to study their affect on MCMC convergence.