Using Search Test Framework and Human Relevance Testing to Evaluate Search Engine Changes for Impact and Relevance Improvements
Edward Chatelain, Khanh Vu, Doug Rosenoff, Tara Diedrichsen · SSRN Electronic Journal · 2019
Search Test Framework (STF) is a state-of-the-art tool for quantifying and comparing search relevance results. In this work, we cover how to use STF, including logging into the tool, creating Singleton, Comparison and Regression tests, and viewing the test results. After covering the basics on how to use STF, we focus on how to use STF to see changes to impact and relevance using a baseline set of queries which have been evaluated in a Human Relevance Test (HRT) study with DCG-Human comparison test results. Finally, we walk through the process that was used to analyze the results of a successful algorithm update.