Building a Full-text Search Index Using "Transformer" Neural Network
Vyacheslav Dobrynin, Roman Abramovich, Alexey Platonov · 2023
The use of deep neural networks in information retrieval significantly improves its effectiveness, but negatively affects the performance of the process. To deal with this, we propose a new ranking model that uses the deep neural network of the "Transformer" architecture (in particular, BERT) for efficient information retrieval. In accordance with the proposed approach, contextualized vector representations are extracted from documents during indexing, after which these representations are clustered for each independent token. The resulting clusters reflect different meanings of the words and are indirectly used as inverted index keys. The values represent the documents in which these contextualized word meanings occur, along with the distances from each document to the contextualized embedding. Thus, after the indexing process, we obtain an index containing pre-calculated distances between the contextualized meanings of dictionary elements and documents. This approach helps us avoid the performance overhead of calculating distances online. At the search stage, the query is transformed into a set of contextualized vectors representing each query token, which allows us to use these vectors to retrieve most semantically close neighbor-tokens and use them to extract relevant documents from the index. This way of searching for contextualized embeddings consumes less memory and is more performant due to the use of an inverted index.