Isotropy Matters: Soft-ZCA Whitening of Embeddings for Semantic Code Search
Andor Diera, Lukas Paul Achatius Galke, Ansgar Scherp · 2025
Low isotropy in an embedding space impairs performance on tasks involving semantic inference.Our study investigates the impact of isotropy on semantic code search performance and explores post-processing techniques to mitigate this issue.We analyze various code language models, examine isotropy in their embedding spaces, and its influence on search effectiveness.We propose a modified ZCA whitening technique to control isotropy levels in embeddings.Our results demonstrate that Soft-ZCA whitening improves the performance of pre-trained code language models and can complement contrastive fine-tuning.