Forging Better Axes: Evaluating and Improving the Measurement of Semantic Dimensions in Word Embeddings
Andrei Boutyline, Ethan Johnston · 2023
Word embeddings are a powerful tool for measuring cultural meaning using large text corpora. In sociology, some of their most common applications estimate relationships between concepts like intelligence or disability and latent semantic axes like status or stigma. These techniques, however, are underdeveloped, with no metrics to capture axis quality and no clear understanding of what properties this quality entails. We tackle these issues by linking this task to the better-studied problem of solving analogies. This lets us identify properties relevant to axis reliability--parallelism, antonymy, and synonymy--and construct metrics to measure them. We then employ existing survey data with ratings of several hundred target terms along 23 semantic dimensions to calculate the accuracy of 23,000 axes that operationalize these 23 dimensions in embedding space. We use these accuracy ratings to evaluate our metrics. Finally, to demonstrate how our metrics can be applied in empirical research, we use them to improve a semantic axis studied in prior work.