Forging Better Axes: Evaluating and Improving the Reliability of Semantic Dimensions in Word Embeddings
Andrei Boutyline, Ethan Johnston · 2025
Word embeddings are a powerful tool for measuring cultural meaning using large text corpora. In sociology, common applications estimate the meanings of concepts by projecting keywords onto latent semantic axes constructed from pairs of opposing anchor terms. This task requires researchers to identify anchor pairs that reliably operationalize the semantic dimension. However, the methodological toolkit for doing this remains underdeveloped—there are no established metrics for assessing anchor set reliability from embedding data, and indeed no clear theoretical understanding of what this reliability entails. We address this gap by investigating three forms of reliability potentially relevant to anchor sets: parallelism between offset vectors, synonymy between anchors at the same endpoint, and antonymy between opposed anchors. Using an original N=1750 survey dataset of respondents’ ratings of several hundred terms along 36 semantic dimensions, plus existing ratings for 3 additional dimensions, we evaluate how well these reliability metrics predict the accuracy of 39,000 axes that operationalize these dimensions in embedding space. We find that parallelism robustly outperforms the other metrics in predicting accuracy. We demonstrate several ways researchers can use parallelism to improve their anchor sets, including identifying problematic anchors and suggesting promising antonyms. We conclude with broader recommendations for semantic axis construction.