Leveraging Automated Search Relevance Evaluation to Improve System Deployment
Yizhao Ni, Ferosh Jacob, Priya Gopi Achuthan, Hui Wu, Faizan Javed · Proceedings of the 31st ACM International Conference on Information & Knowledge Management · 2022
Over the last year, a digital initiative has been focused on reengineering the search engine for kp.org, a health web portal serving over 12 million members. However, traditional software testing techniques that rely on limited use cases and consistent behavior are neither comprehensive nor specific for capturing complex user search behaviors. To support system deployment, we utilize information retrieval (IR) technologies to monitor search performance, identify areas of improvement and suggest actionable items. In this case study we share industrial experience on building an IR evaluation pipeline and its usage to inform deployment and improve system development. The work emphasizes domain specific challenges, best practices and lessons learned during system deployment in a healthcare setting. It features the ability of IR techniques to strengthen collaboration between data scientists, software engineers and product managers in making data-driven decisions.