Syntactic dependencies correspond to word pairs with high mutual information
Richard Futrell, Peng Qian, Edward A. F. Gibson, Evelina G. Fedorenko, Idan Blank · 2019
How is syntactic dependency structure reflected in the statistical distribution of words in corpora?Here we give empirical evidence and theoretical arguments for what we call the Head-Dependent Mutual Information (HDMI) Hypothesis: that syntactic heads and their dependents correspond to word pairs with especially high mutual information, an information-theoretic measure of strength of association.In support of this idea, we estimate mutual information between word pairs in dependencies based on an automatically-parsed corpus of 320 million tokens of English web text, finding that the mutual information between words in dependencies is robustly higher than a controlled baseline consisting of non-dependent word pairs.Next, we give a formal argument which derives the HDMI Hypothesis from a probabilistic interpretation of the postulates of dependency grammar.Our study also provides some useful empirical results about mutual information in corpora: we find that maximum-likelihood estimates of mutual information between raw wordforms are biased even at our large sample size, and we find that there is a general decay of mutual information between part-of-speech tags with distance.