Multi-resolution Odd Sketch for Mining Jaccard Similarities between Dynamic Streaming Sets
Qingjun Xiao, Lin Wen, Quanwei Zhang · 2021
Estimating similarity between streaming sets is a fundamental problem with many Internet applications, such as evaluating user similarity in social networks and analyzing similarity of IP hosts' behaviors in communication networks. For a “streaming” data set, its elements arrive in a streaming fashion, and we have only limited memory to process its element stream. To meet the size constraint of high-speed memory, this data set must be stored as a summary called `sketch'. Then, in the distributed scenario, the sketches of all data sets can be efficiently transferred to a central server to calculate the Jaccard similarity between each pair of sets. To balance between memory cost and similarity evaluation accuracy, many sketching methods have been proposed, such as MinHash, virtual odd sketch (VOS) and MaxLogHash. However, both MinHash and MaxLogHash fail to deal with fully dynamic streaming sets that allow the deletion of elements. Although VOS partially solves the deletion problem by adopting the odd sketch structure and enhance it with a physical-virtual structure, its similarity evaluation accuracy will degrade when handling small streaming sets. In this paper, we propose a multi-resolution odd sketch (MROS), which allows more accurate similarity estimation with less memory consumption. Its design is to encode a streaming set into multiple layers of odd sketches with exponentially reducing sampling probabilities. We conduct both experiments and analysis to evaluate our method. Results show that the estimation accuracy of our MROS outperforms existing works, e.g., MinHash and VOS.