Will the user engage with the item? A probabilistic framework for personalized product recommendation in big data environment
Simona‐Vasilica Oprea, Adela Bârã · Big Data Research · 2026
This paper introduces a classification-based recommendation system that outputs purchase probabilities rather than just recommending items based on traditional methods. Unlike traditional methods that focus on rating prediction or implicit feedback, this system provides a probabilistic ranking that is directly used for personalized marketing strategies, such as targeted price discounts. By leveraging large-scale datasets (1.2 billion records) and six machine learning (ML) classifiers, the proposed system is designed to be scalable to e-commerce environments. Additionally, our research compares the performance of the classification-based system against classical collaborative filtering techniques using the Surprise package. Furthermore, the use of different data formats including JSON-based order data highlights the system’s ability to handle heterogeneous and unstructured data sources. The recommender system is updated as the users, items and orders update dynamically. The high volume, velocity and variety of data position our probabilistic approach within the framework of big data. Two strategies are foreseen to handle big data: data sampling or applying XGBoost and LightGBM with parallelism, GPU acceleration and out-of-core training. The proposal of a dynamic pricing adjustment based on predicted purchase probabilities (e.g., discounts for items with probability between 0.35 and 0.49) bridges the gap between predictive modeling and actionable business decisions.