Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am learning about the maths behind word2vec from this tutorial.
u are the embeddings for the center word and v for the context. It appears that this dot product is to be maximized. Why the context words are not necessarily similar to the center. Sure they appear together, but it does not mean they are synonyms.
Where the objective function of Skip-Gram Negative Sampling (SGNS) come from? is effectively a more general version of your question. My answer there should answer your question.
In short, word2vec learns to classify which words (called context words) appear around a particular (target) word from those that don't. The dot product is just a simple model design choice, like choosing a simple linear regression model.
This prediction task is actually a slight red herring as the main aim is to learn useful word embeddings. It just turns out that training that model to be good at that classification task gives good word embeddings. That is not theoretically obvious, but followed from empirical observations that vectors taken from language models gave good embeddings with interesting semantic structure [1]. Recent work aims to explain why word2vec embeddings have this structure [e.g. 2, 3].
[1] Linguistic Regularities in Continuous Space Word Representations (https://www.aclweb.org/anthology/N13-1090.pdf)
[2] Analogies Explained: Towards Understanding Word Embeddings (http://proceedings.mlr.press/v97/allen19a.html)
[3] What the Vec? Towards Probabilistically Grounded Embeddings
(http://papers.nips.cc/paper/8965-what-the-vec-towards-probabilistically-grounded-embeddings.pdf)
