Building a Multitenant Data Hub System using Elastic Stack and Kafka for Uniform Data Representation
Nebojsa Kuduz, Saša Salapura · 2020
The goal of this paper is to present one way of building and implementing of an Elastic Stack based System for processing data of various types and formats, events from custom applications and for transforming input data to be presented in a uniform way. For practical data hub system implementation, events and logs are collected from various IoT devices through LoRa WAN gateways, messaging gateways and management information subsystems. Data are shipped directly or using Beats. Logstash and Apache Kafka are used for processing and buffering of data streams, to execute different filters, for input data enhancements and transformations and for indexing and storing data into Elasticsearch. Kibana provides visualization capabilities on top of the content indexed by Elasticsearch. System is designed and configured as elastic multitenant solution to target different consumers by indices separated to different spaces and with role-based and attribute-based access controls.