Compact Indexing and Judicious Searching for Billion-Scale Microblog Retrieval
Dongxiang Zhang, Liqiang Nie, Huanbo Luan, Kian‐Lee Tan, Tat‐Seng Chua, Heng Tao Shen · ACM Transactions on Information Systems · 2017
In this article, we study the problem of efficient top-kdisjunctive query processing in a huge microblog dataset. In terms of compact indexing, we categorize the keywords into rare terms and common terms based on inverse document frequency (idf) and propose tailored block-oriented organization to save memory consumption. In terms of fast searching, we classify the queries into three types based on term category and judiciously design an efficient search algorithm for each type. We conducted extensive experiments on a billion-scale Twitter dataset and examined the performance with both simple and more advanced ranking functions. The results showed that with much smaller index size, our search algorithm achieves a factor of 2--3 times faster speedup over state-of-the-art solutions in both ranking scenarios.