A Distributed Recommendation Platform for Big Data
Daniel Valcarce, Javier Parapar, Álvaro Barreiro · Zenodo (CERN European Organization for Nuclear Research) · 2020
Abstract: The vast amount of information that recommenders manage these days has reached a point where scalability has become a critical factor. In this work, we propose a scalable architecture designed for computing Collaborative Filtering recommenda-tions in a Big Data scenario. In order to build a highly scalable and fault-tolerant platform, we employ fully distributed systems without any single point of failure. We study the use of data replication and data distribution technologies. Additionally, we consider different caching techniques. Taking into account these requirements, we pro-pose particular technologies for each component of the platform. Next, we evaluate the response times of storing, generating and serving recommendations using MySQL Cluster and Cassandra showing that the latter technology is much more adequate for that purpose. Finally, we conduct a simulation for evaluating the impact of a memory caching system.