PilotANN: Memory-Bounded GPU Acceleration for Vector Search
Yuntao Gui, Peiqi Yin, Yan Xiao, Chaorui Zhang, Weixi Zhang, James Sheung-Chak Cheng · 2026
Approximate Nearest Neighbor Search (ANNS) has become fundamental to modern deep learning applications, having gained particular prominence through its integration into recent generative models that work with increasingly complex datasets and higher vector dimensions. Existing CPU-only solutions, even the most efficient graph-based ones, struggle to meet these growing computational demands, while GPU-only solutions face memory constraints. As a solution, we propose PilotANN, a hybrid CPU-GPU system for graph-based ANNS that utilizes both CPU's abundant RAM and GPU's parallel processing capabilities. Our key innovation lies in decomposing the top-k search process into three complementary stages of increasing precision and decreasing computational cost: (i) GPU-accelerated subgraph traversal using SVD-reduced vectors; (ii) CPU refinement; and (iii) precise search using complete vectors. Furthermore, we introduce fast entry selection to improve search starting points while maximizing GPU utilization. Experimental results demonstrate that PilotANN achieves 3.9 -- 5.4× speedup in throughput on 100-million scale datasets, and is able to handle datasets up to 12 × larger than the GPU memory.