Divide and Conquer: Towards Better Embedding-based Retrieval for Recommender Systems from a Multi-task Perspective
Yuan Zhang, Xue Dong, Weijie Ding, Biao Li, Peng Jiang, Kun Gai · 2023
Embedding-based retrieval (EBR) methods are widely used in modern recommender systems thanks to its simplicity and effectiveness. However, along the journey of deploying and iterating on EBR in production, we still identify some fundamental issues in existing methods. First, when dealing with large corpus of candidate items, EBR models often have difficulties in balancing the performance on distinguishing highly relevant items (positives) from both irrelevant ones (easy negatives) and from somewhat related yet not competitive ones (hard negatives). Also, we have little control in the diversity and fairness of the retrieval results because of the “greedy” nature of nearest vector search. These issues compromise the performance of EBR methods in large-scale industrial scenarios.