A Manifold Learning Method to Passage Retrieval for Open-Domain Question Answering
Ruidong Ding, Bin Zhou, Hongkui Tu · 2021
Passage retriever plays an important role for obtaining answers in open-domain textual question answering system, which selects candidate contexts from a large collection of documents and feed to the machine reader. Traditional defacto methods usually construct sparse vectors to match the rules of co-occurrence of words between passages and questions, such as TF-IDF or BM25. And some more advanced methods model word-level contextual semantics similarities to match the text. In this work, we presents a method of encoding text by short sliding windows with built-in continuity, and applying manifold learning method on it to model continuous representation of semantics, so as to represent the similarity features at the passage-level and reduce the directional sparsity difference caused by the difference of text length. Compared with the traditional Lucene BM25 system in the top-20 paragraphs retrieval, the accuracy of our method is 5%-16% higher, and the recall rate is 8%-16% higher.