How word embedding work for word similarity?

Question

I am trying to apply word2vec/doc2vec to find similar sentences. First consider word2vec for word similarity. What I understand is, CBOW can be used to find most suitable word given a context, whereas Skip-gram is used to find the context given some word, so in both cases, I am getting words that co-occur frequently. But how does it work to find similar words? My intuition is, since similar words tend to occur in similar contexts, the words similarity is actually measured from the similarity among contextual/co-occuring words. In the neural net, when the vector representation for some word at the hidden layer is passed through to the output layer, it outputs probabilities of co-occuring words. So, the co-occuring words influence the vectors of some words, and since similar words have similar set of co-occuring words, their vector representations are also similar. To find the similarity, we need to extract the hidden layer weights (or vectors) for each word and measure their similarities. Do I understand it correctly?

Finally, what is a good way to find tweet text (full sentence) similarity using word2vec/doc2vec?

Answer 1

I think you have it mostly correct.

Word embeddings can be summed up by: A word is known by the company it keeps. You either predict the word given the context or vice versa. In either case similarity of word vectors is similarity in terms of replaceability. i.e. if two words are similar one could replace the other in the same context. Note that this means that "hot" and "cold" are (or might be) similar within this context.

If you want to use word embeddings for a similarity measure of tweets there are a couple approaches you can take. One is to compute paragraph vectors (AKA doc2vec) on the corpus, treating each tweet as a separate document. (There are good examples of running doc2vec on Gensim on the web.) An alternate approach is to AVERAGE the individual word vectors from within each tweet, thus representing each document as an average of its word2vec vectors. There are a number of other issues involved in optimizing similarity on tweet text (normalizing text, etc) but that is a different topic.