SynEvaRec: A Framework for Evaluating Recommender Systems on Synthetic Data Classes
Vladimir Provalov, Elizaveta Stavinova, Petr Chunaev · 2021 International Conference on Data Mining Workshops (ICDMW) · 2021
This study proposes a novel method for evaluating and comparing recommender systems using synthetic user and item data and parametric synthetic user-item response (rating) functions. The method compares recommender systems on classes of synthetic data, oppositely to how it is usually done on particular real or synthetic datasets. The usage of classes particularly allows for managing the effects of the No Free Lunch theorem for recommender systems. Furthermore, we implement the method in the form of a flexible framework (that we call SynEvaRec) for conducting comparison experiments under different scenarios of synthetic data behaviour. Our experimental study shows that SynEvaRec helps to determine scenarios (e.g. in terms of data classes) where one recommender system is more preferable than another by means of recommendation quality. Moreover, the results turn to be rather stable over several synthetic dataset instances based on the same real-world dataset indicating the robustness of our method. The datasets, the framework implementation and the results related to our study are publicly available on GitHub.