A managed distributed processing pipeline with Storm and Mesos
Dragos Dena, Mihai Bucicoiu, Mircea Bardac · 2013
Running an always-on distributed system that has the task of continuous message processing can raise a series of problems like resource sharing with the other services running in the cluster, controlling the locality of the worker processes and the ability to move workers from one machine to another without impacting the functionality of the system or inducing data loss. In this paper are making an in-depth analysis of these problems and design a system that addresses them. Our computation pipeline takes the form of a Storm topology and with dynamic resource management and isolation for it being achieved through Apache Mesos. The focus is on isolating and distributing resources between the nodes of one topology instance and other services.