Which Vertical Search Engines are Relevant? Understanding Vertical Relevance Assessments for Web Queries
Ke Zhou, Ronan Cummins, Mounia Lalmas, Joemon M. Jose · 2013
Aggregating search results from a variety of heterogeneous sources, so-called verticals, such as news, image and video, into a single interface is a popular paradigm in web search. Current approaches that evaluate the effectiveness of aggregated search systems are based on rewarding systems that return highly relevant verticals for a given query, where this relevance is assessed under different assumptions. It is difficult to evaluate or compare those systems without fully understanding the relationship between those underlyingassumptions. To addressthis, we presentaformal analysis and asetofextensiveuserstudiestoinvestigatetheeffectsofvarious assumptionsmadefor assessing queryverticalrelevance. A total of more than 20,000 assessments on 44 search tasks across 11 verticals are collected through Amazon Mechanical Turk and subsequently analysed. Our results provide insights into various aspects of query vertical relevance and allow us to explain in more depth as well as questioning the evaluation results published in the literature.