1
$\begingroup$

I am trying to apply word2vec/doc2vec to find similar sentences. First consider word2vec for word similarity. What I understand is, CBOW can be used to find most suitable word given a context, whereas Skip-gram is used to find the context given some word, so in both cases, I am getting words that co-occur frequently. But how does it work to find similar words? My intuition is, since similar words tend to occur in similar contexts, the words similarity is actually measured from the similarity among contextual/co-occuring words. In the neural net, when the vector representation for some word at the hidden layer is passed through to the output layer, it outputs probabilities of co-occuring words. So, the co-occuring words influence the vectors of some words, and since similar words have similar set of co-occuring words, their vector representations are also similar. To find the similarity, we need to extract the hidden layer weights (or vectors) for each word and measure their similarities. Do I understand it correctly?

Finally, what is a good way to find tweet text (full sentence) similarity using word2vec/doc2vec?

$\endgroup$
2
  • $\begingroup$ I wanted to achieve something similar. What I have is the word embedding of two paragraphs and I wish to find out how close they are in terms of their context. I had a approach in mind, although I am highly doubtful of its practicality. What I thought of was, given two word embeddings(of two paragraphs), if I could calculate something analogus to a distance score as means of a similarity measure, then that would tell me how close each of the candidate paragraphs are to my anchor point. It would be great if I could get some help on that. Thank you for your time! $\endgroup$ Dec 10, 2018 at 5:07
  • $\begingroup$ If you have a new question, please ask it by clicking the Ask Question button. Include a link to this question if it helps provide context. - From Review $\endgroup$
    – oW_
    Dec 10, 2018 at 19:48

1 Answer 1

2
$\begingroup$

I think you have it mostly correct.

Word embeddings can be summed up by: A word is known by the company it keeps. You either predict the word given the context or vice versa. In either case similarity of word vectors is similarity in terms of replaceability. i.e. if two words are similar one could replace the other in the same context. Note that this means that "hot" and "cold" are (or might be) similar within this context.

If you want to use word embeddings for a similarity measure of tweets there are a couple approaches you can take. One is to compute paragraph vectors (AKA doc2vec) on the corpus, treating each tweet as a separate document. (There are good examples of running doc2vec on Gensim on the web.) An alternate approach is to AVERAGE the individual word vectors from within each tweet, thus representing each document as an average of its word2vec vectors. There are a number of other issues involved in optimizing similarity on tweet text (normalizing text, etc) but that is a different topic.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.