Towards Personalized Recommender System: A Gray-Box Modeling Approach
Mary Nusrat, Gahangir Hossain · 2025
As the volume of big data continues to expand, the importance of Recommender Systems (RS/RecSys) has become increasingly recognized for their ability to focus user search on relevant information or products, rather overwhelming users with irrelevant options. This paper presents a design of a personalized recommender engine using model-based collaborative filtering using gray-box explainable transparent system, comparing the performance of two widely used machine learning algorithms: K-Nearest Neighbor (KNN) and Alternating Least Square (ALS). KNN applies cosine similarity between user vectors. However, ALS, inspired by the success of the Netflix Prize recommender engine, implements an embedding-based approach and is deployed using Apache Spark for efficient distributed computation, leveraging powerful SQL databases and machine learning libraries. Common issues encountered with KNN-based RS were mitigated using ALS. In the context of the personalized recommender engine described in this work, the hybridization of KNN and ALS can be seen as foundational elements of a gray-box approach for better explainability. While we focused on KNN and ALS, other deep learning-based embedding like Neural Matrix Factorization (NMF) and Graph-based Neural Networks (GNN) were also explored but excluded due to their lack of transparency and explainability in the recommendation process, which can hinder user trust and understanding. Performance was evaluated using RMSE and MAE metrics.