Evaluating search interfaces using real-time emotional and behavioral data
Abbas Pirmoradi Bezanjani · 2025
Interactive Information Retrieval (IIR) interfaces are often evaluated using post-task questionnaires that gather subjective measures such as satisfaction, ease of use, usefulness, and user engagement, as well as in-task objective measures obtained from log analysis. While these methods provide valuable insights, a more comprehensive evaluation requires a deeper understanding of user behaviour. This thesis aims to bridge this gap by developing a comprehensive framework that integrates real-time emotional responses with behavioural data, enabling a more nuanced evaluation of search interfaces. In the first study, we developed a novel method for capturing real-time emotional responses using consumer-grade front-facing cameras. This approach synchronized emotional data with user interactions to assess how search result relevance influences emotional responses, post-task evaluations, and feature interactions. The study, which focused on exploratory search in a controlled laboratory environment, revealed that both positive and negative emotional responses could be reliably detected during the search process, with clear evidence of recency effects impacting post-task evaluations. The study also identified specific interactive features that correlated with these emotional responses, providing a foundation for integrating real-time emotions into IIR evaluations. The second study expanded the framework’s application to a novel academic digital library search interface, Tag Search Results (TagSR), which included advanced features like dynamic tagging, highlighting, and filtering aimed at enhancing exploratory search. Comparing TagSR to a Baseline interface in a controlled lab study demonstrated that the advanced features significantly improved perceived value, including ease of use, usefulness, satisfaction, and user enigagement. Also, we found that the precision of saved search results increased without increasing task completion time, highlighting the benefits of incorporating these advanced feature in search interfaces. Using the approach developed in the first study, in-task evaluation of these interfaces emphasized how emotional responses to specific interface features can provide deeper insights into user behaviour and experience during the search process. In the third study, we extended the framework to include eye tracking data allowing for the analysis of emotional responses around features that are viewed but not used. The study, conducted in a controlled laboratory setting, focused on exploratory search within digital humanities repositories. Integrating eye-tracking data with logged feature use and facial emotion expressions provided a holistic approach to evaluating these search interfaces at the feature level. Through this approach, we were able to evaluate not only traditional interactions with search interfaces (looking at features, using them, and experiencing an emotional reaction) but also passive interactions with search interfaces (looking at features, choosing not to use them, but possibly gaining information from them, and experiencing emotional reactions). This approach allowed us to identify specific features of the interfaces that generated positive and negative emotional responses when used, as well as those that generated such emotional responses when viewed but not used. It would be difficult to capture such feature-level observations using other methods, which would provide insight into how searchers interact with search interfaces. Overall, this thesis presents a robust framework for incorporating emotional and behavioural data into the evaluation of IIR interfaces. By demonstrating that real-time emotional responses can be effectively integrated with traditional metrics, this work provides methods for developing deeper insights into user behaviour, offers a more holistic evaluation approach, and suggests innovative directions for evaluating search interfaces in academic, digital humanities, and other content-rich domains.