Second-order Co-occurrence Sensitivity of Skip-Gram with Negative Sampling
Dominik Schlechtweg, Cennet Oğuz, Sabine Schulte im Walde · 2019
We simulate first-and second-order context overlap and show that Skip-Gram with Negative Sampling is similar to Singular Value Decomposition in capturing second-order cooccurrence information, while Pointwise Mutual Information is agnostic to it.We support the results with an empirical study finding that the models react differently when provided with additional second-order information.Our findings reveal a basic property of Skip-Gram with Negative Sampling and point towards an explanation of its success on a variety of tasks.