An extensible Hadoop framework for monitoring performance metrics and events of OpenStack cloud
Jianxi Yang, Chaoxiao Shen, Yaping Chi, Ping Xu, Wei Sun · 2018
In recent years, many hot topics for research are manipulation of big data, cloud computing and a combination of them. OpenStack is a complex and evolving system that continuously generates vast amounts of metrics and log data. The more complex your system gets, the harder it is to understand its performance and troubleshoot problems, making monitoring a critical piece of the OpenStack control system. In order to effectively create, test and deploy new algorithms or frameworks one need suitable monitoring solutions. However, the conventional monitoring tools lack some performances to keep data freshness from a variety of sources in OpenStack. In the paper we design a critical monitoring solutions based on existing Hadoop architecture integrated with OpenStack clouds. Hundreds of different metrics are gathered form Hadoop metrics subsystem and all data is stored in HBase database to make it ready for processing or displaying it with MapReduce paradigm. With alarms/events generated, we also present a comprehensive view of the status of the infrastructure resources as well as the network services running on OpenStack.