Message-oriented Middleware for Scalable Data Analytics Architectures
Nicolas Nannoni · KTH Publication Database DiVA (KTH Royal Institute of Technology) · 2015
The democratization of Internet allowed many more people to use online services and enjoy their benefits. The traffic towards websites became tremendous those recent years, especially with the apparition of social networks. Mobile application, televisions and other non--‐computer devices also get connected to the Internet and use it to provide services to the end--‐users: Video on--‐demand, music streaming and so on. These applications rely on powerful backend servers that handle the requests made by devices and provide statistics and metrics about application usage. These metrics can be generated by aggregating the access logs (e.g. HTTP requests log), logs that are potentially extremely large. Big data tools and analytics, providing a way to handle this huge number of records, come then in hand, as typical client--‐server architectures, with a single database storing all the data, reach their limits in terms of performance and capacity. Data duplication, combined to dedicated and specialized databases storing it, is the key to efficient data handling. How to fill up these databases in an elegant, efficient and scalable manner is the remaining question, and message--‐oriented middleware may be a viable answer. This project aims at exploring the capabilities of such middleware, identifying what are the benefits and the drawbacks in using them and presenting how they can be integrated in a real--‐world application that needs to aggregate events and logs on a large scale. Apache Kafka and RabbitMQ, two message--‐oriented middleware, are benchmarked and compared, on both performance metrics and qualitative criteria. A fully working proof--‐ of--‐concept (of an already--‐existing industry product modified to use a message--‐oriented middleware and a specialized data warehouse system) is developed and presented, to conclude on the usefulness of message--‐oriented middleware when designing scalable data analytics architectures.