Taking advantage of spontaneous speech for document retrieval: Lessons from the organization of evaluation tasks
Tomoyosi Akiba · The Journal of the Acoustical Society of America · 2016
This work reports our experiments on organizing evaluation tasks aiming at spoken document retrieval (SDR). The NTCIR Project is a series of evaluation workshops designed to enhance research in information access technologies by providing reusable test collections and a forum for researchers. The author's group has proposed the evaluation tasks for SDR at the NTCIR from 2010. They resulted in holding the four successive evaluation tasks called SpokenDoc-1 and 2, and SpokenQuery&Doc-1 and 2. In the SpokenDoc tasks conducted from 2010 to 2013, we designed and evaluated the SDR tasks targeting spontaneously spoken speech. Based on this data collection, we conducted two subtasks: the spoken term detection subtask and the spoken content retrieval subtask. After two rounds of them, we enhanced our research focus with the query part of information retrieval. In the SpokenQuery&Doc tasks conducted from 2013 to 2016, we investigated the information retrieval systems that take advantage of a spontaneously spoken query as a medium of expressing one's information need. We have collected spontaneously spoken queries through subjective experiment to construct search topics for information retrieval. All of those experiences has been compiled into the reusable test collections provided at the NTCIR project, which enables researchers to test their ideas anytime on the common testbeds. In this presentation, we will report our task design, methodologies and challenges on constructing our data, and outcomes found through our organization.