Optimization of All Pairs Similarity Search
Yuechen Chen, Xinhuai Tang, Bing Liu, Delai Chen · 2015
All pairs similarity search (APSS) is the problem of finding all the similar pairs of items, whose similarity is above a given threshold. APSS algorithm is applied to many data mining fields, such as document matching, collaborative filtering. Due to a large scale of data in real life, some recent work used partitioning, inverted indexing, parallel accumulation, and hashing approximation to optimize the APSS algorithm. To optimize the APSS problem, this paper analyzes and compares two parallel approaches. To demonstrate the performance gain of our optimization approaches, we implement our algorithms on Spark and conduct the evaluation on a dataset of one million movies, which gains better performance speedup than other works.