Semi-sparse algorithm based on multi-layer optimization for recommendation system
Hu Guan, Huakang Li, Minyi Guo · 2012
Collaborative filter (CF) is the most successful technique in recommender system, which makes personalize recommendations during online interaction. We propose a new Semi-sparse algorithm based on multi-layer optimization to speed up the basic Pearson Correlation Coefficient of CF. Semi-sparse algorithm spares out over-reduplicate accessing and judgement on selected sparse vector to accelerate the batch of similarity-comparisons in one thread. We propose a reduce-vector in thread-pool to restrict the lock using on critical resources in parallelize implementation. Thread-pool is wrapped with Pthreads on multi-core node to make semi-sparse parallelization more easily. A shared zip file is read to cut down messages with Message Passing Interface package. The performance of proposed semi-sparse with multi-layer framework achieved a brilliant speedup in the evaluation of Netflix, MovieLens and MovieLen1600.