Weighted Edit Distance based FAQ Retrieval using Noisy Queries
Shahbaaz Mhaisale, Sangameshwar Patil, Kiran Mahamuni · 2013
In this paper, we describe our contribution to the FIRE 2013 shared task on "FAQ Retrieval using Noisy Queries". Short messaging service (SMS) and voice-based interfaces such as Siri have become quite popular for quick information retrieval these days. The problem posed in this FIRE 2013 shared task is that given an SMS query or speech-to-text transcript of a voice based query, find the best matching frequently asked question (FAQ) from a database or return NULL if there is no relevant FAQ. In our solution, we first normalize the query and then compute the similarity between the query and each question using weighted edit distance. We observe that instead of using the standard edit distance which gives equal weight for all the string edit operations (such as insertion, deletion, transposition, substitution), weighted edit distance works significantly better for checking similarity of SMS queries with FAQ database questions. We determine whether a query is out of domain (i.e. cannot be answered using current set of questions in the FAQ database) using a simple classifier using FAQ database itself. Our method identified 105 out of 148 out-of-domain queries correctly and 188 out of 392 in-domain queries correctly with mean reciprocal rank (MRR) of 0.8836. Since we observed very little correlation in the transcript of speech queries with best matching question in the training data, we also report the results for SMS only queries. On the SMS queries, our method identified 188 of 200 queries correctly, resulting in an MRR of 0.968.