In CBOW, each word has input vector and output vector, or context vector and self vector, the context aggregate the surrounding words, what if the context also contain the input vector of the center word?
1 Answer
$\begingroup$
$\endgroup$
Both CBOW and Skipgram models (from the word2vec paper) consider each (target) word in turn together with nearby words within a fixed window (context words). The CBOW model learns to predict the target from its surrounding context words. Skipgram does the opposite.
Ideally, any word embeddings of similar words should have similar embeddings. That happens in word2vec embeddings because similar target words have similar distributions of context words around them and so similar parameters (i.e. the embeddings) predict them, e.g. if dog appears with walk, tail, kennel, etc and hound also appears with those words, then dog and hound have similar embeddings.
If the target word is included in the distribution of context words (dog in the context words of dog, hound in those of hound) they occur in the context distribution every time the target word appears, rather then some times like other context words, so have a dominant effect. That creates a difference between the two context distributions and so a difference between the embeddings, which probably then perform worse at tasks that require similar words to have similar embeddings.
