MIXED-LSH: Reduction of Remote Accesses in Distributed Locality-Sensitive Hashing Based on L1-distance
Hisashi Koga, Masayuki Oguri, Toshinori Watanabe · 2012
Locality-Sensitive Hashing (LSH) is a well-known approximate nearest-neighbor search algorithm for high-dimensional data. Though LSH searches nearest-neighbor points for a query very fast, LSH has a drawback that the space complexity is very large. For this reason, so as to apply LSH to a large dataset, it is crucial to implement LSH in distributed environments which consist of multiple nodes. One simple and natural method to implement LSH in the distributed environment is to have every node keep the same number of hash tables. However, this method increases remote accesses, because many nodes are accessed to access all the hash tables. Thus, this simple method will suffer from the long query response time, if the communication delay is the bottleneck. This paper proposes to reduce remote accesses by assigning hash buckets smartly to the nodes. In particular, our method assigns hash buckets from different hash tables to the same node, if the hash buckets store the same points. Due to this strategy, our method can access multiple hash buckets that should be accessed in processing a query with a single remote access, thereby decreasing remote accesses.