Investigating Crowdsourcing as an Evaluation Method for TEL Recommenders.
Mojisola Erdt, Florian Jomrich, Katja Schüler, Christoph Rensing · 2013
Offline evaluations using historical data offer a fast and repeatable way to evaluate TEL recommender systems. However, this is only possible if historical datasets contain all particular information needed by the recommender algorithm. Another challenge is that users must have indicated interest in the recommended resource in the past for a resource to be evaluated as relevant. This however does not mean the user would not be interested in this newly recommended resource. User experiments help to complement offline evaluations but due to the effort and costs of performing these experiments, very few are conducted. Crowdsourcing is a solution to this challenge as it gives access to sufficient willing users. This paper investigates the evaluation of a graphbased recommender system for TEL using crowdsourcing. Initial results show that crowdsourcing can indeed be used as an evaluation method for TEL recommender systems.