SeaRank: relevance prediction based on click models in a reinforcement learning framework
Amir Hosein Keyhanipour, Farhad Oroumchian · Data Technologies and Applications · 2022
Purpose User feedback inferred from the user's search-time behavior could improve the learning to rank (L2R) algorithms. Click models (CMs) present probabilistic frameworks for describing and predicting the user's clicks during search sessions. Most of these CMs are based on common assumptions such asAttractiveness, ExaminationandUser Satisfaction. CMs usually consider theAttractivenessandExaminationas pre- and post-estimators of the actual relevance. They also assume thatUser Satisfactionis a function of the actual relevance. This paper extends the authors' previous work by building a reinforcement learning (RL) model to predict the relevance. TheAttractiveness, ExaminationandUser Satisfactionare estimated using a limited number of the features of the utilized benchmark data set and then they are incorporated in the construction of an RL agent. The proposed RL model learns to predict the relevance label of documents with respect to a given query more effectively than the baseline RL models for those data sets. Design/methodology/approach In this paper,User Satisfactionis used as an indication of the relevance level of a query to a document.User Satisfactionitself is estimated throughAttractivenessandExamination, and in turn,AttractivenessandExaminationare calculated by the random forest algorithm. In this process, only a small subset of top information retrieval (IR) features are used, which are selected based on their mean average precision and normalized discounted cumulative gain values. Based on the authors' observations, the multiplication of theAttractivenessandExaminationvalues of a given query–document pair closely approximates theUser Satisfactionand hence therelevance level. Besides, an RL model is designed in such a way that the current state of the RL agent is determined by discretization of the estimatedAttractivenessandExaminationvalues. In this way, each query–document pair would be mapped into a specific state based on itsAttractivenessandExaminationvalues. Then, based on the reward function, the RL agent would try to choose an action (relevance label) which maximizes the received reward in its current state. Using temporal difference (TD) learning algorithms, such as Q-learning and SARSA, the learning agent gradually learns to identify an appropriate relevance label in each state. The reward that is used in the RL agent is proportional to the difference between theUser Satisfactionand the selected action. Findings Experimental results on MSLR-WEB10K and WCL2R benchmark data sets demonstrate that the proposed algorithm, named as SeaRank, outperforms baseline algorithms. Improvement is more noticeable in top-ranked results, which usually receive more attention from users. Originality/value This research provides a mapping from IR features to the CM features and thereafter utilizes these newly generated features to build an RL model. This RL model is proposed with the definition of the states, actions and reward function. By applying TD learning algorithms, such as the Q-learning and SARSA, within several learning episodes, the RL agent would be able to learn how to choose the most appropriate relevance label for a given pair of query–document.