DimReg: Embedding Dimension Search via Regularization for Recommender Systems
Mingjun Zhao, Liyao Jiang, Yakun Yu, Xinmin Wang, Yuan Yi, Wei Zheng, Di Niu · Society for Industrial and Applied Mathematics eBooks · 2024
Modern recommender systems aim to identify items that are most pertinent to a particular user and are particularly useful when an overwhelming number of items are present. Feature embedding is essential to deep recommender systems, which constructs memory-efficient and semantically meaningful representations by mapping high-dimensional sparse feature vectors into low-dimensional dense vectors. Most existing systems assign a unified dimension to all feature fields, regardless of the diverse importance of different features, which usually results in sub-optimal performance and high memory usage. In this paper, we propose a low-cost embedding dimension search approach named DimReg for recommender systems, by assessing information overlapping between the dimensions within each feature field and pruning unimportant and redundant dimensions progressively during model training via a two-level polarization regularizer, while introducing minimum overhead. Moreover, our method does not require retraining after embedding dimension search, which significantly reduces the computational cost and is more friendly to deployment in real-world recommender systems. Extensive experiments conducted on multiple CTR (Click Through Rate) prediction tasks demonstrate that our method can efficiently reduce the model parameters up to 98.6%, and achieve strong recommendation performance outperforming existing automated embedding dimension search methods.