EvalUMAP: Towards comparative evaluation in user modeling, adaptation and personalization
Owen Conlan, Liadh Kelly, Kevin Koidl, Séamus Lawless, Athanasios Staikopoulos · NORMA · 2017
There is currently no established or standardized means for the comparative evaluation of algorithms and systems developed by researchers in the User Modeling, Adaptation and Personalization (UMAP) space. The design and establishment of such methodologies has proven to be extremely difficult, but would be highly rewarding, as demonstrated by initiatives such as CLEF, TREC and NTCIR in the Information Retrieval domain. Privacy concerns, the challenges of working with interactive scenarios, and individual differences in behaviour between users must all be addressed in order to facilitate repeatable and comparable evaluation, and to advance research in this domain. In this paper we present EvalUMAP, a new concerted drive towards the establishment of shared challenges for comparative evaluation within the UMAP community.