28
$\begingroup$

Given a sentence: "When I open the ?? door it starts heating automatically"

I would like to get the list of possible words in ?? with a probability.

The basic concept used in word2vec model is to "predict" a word given surrounding context.

Once the model is build, what is the right context vectors operation to perform my prediction task on new sentences?

Is it simply a linear sum?

model.most_similar(positive=['When','I','open','the','door','it','starts' ,
                   'heating','automatically'])
Improve this question
$\endgroup$
1
  • $\begingroup$ I implemented the above algorithm and came across a question: Why is softmax used? Let me show you an example of two normalization functions: def softmax(w, t = 1.0): # Source: gist.github.com/stober/1946926 e = np.exp(w / t) return e / np.sum(e) def normalization(w): return w / np.sum(w) a = np.array([.0002, .0001, .01, .03]) print normalization(a) print softmax(a, t=1) Let's compare the outputs: [ 0.00496278 0.00248139 0.24813896 0.74441687] [ 0.24752496 0.24750021 0.24996263 0.25501221] As we can see, softmax gives .03 roughly the same probability as compared to .0001 (which is $\endgroup$
    – user23247
    Commented Aug 9, 2016 at 8:13

2 Answers 2

Reset to default
13
$\begingroup$

Word2vec works in two models CBOW and skip-gram. Let's take CBOW model, as your question goes in the same way that predict the target word, given the surrounding words.

Fundamentally, the model develops input and output weight matrices, which depends upon the input context words and output target word with the help of a hidden layer. Thus back-propagation is used to update the weights when the error difference between predicted output vector and the current output matrix.

Basically speaking, predicting the target word from given context words is used as an equation to obtain the optimal weight matrix for the given data.

To answer the second part, it seems a bit complex than just a linear sum.

  1. Obtain all the word vectors of context words
  2. Average them to find out the hidden layer vector h of size Nx1
  3. Obtain the output matrix syn1(word2vec.c or gensim) which is of size VxN
  4. Multiply syn1 by h, the resulting vector will be z with size Vx1
  5. Compute the probability vector y = softmax(z) with size Vx1, where the highest probability denotes the one-hot representation of the target word in vocabulary. V denotes size of vocabulary and N denotes size of embedding vector.

Source : http://cs224d.stanford.edu/lecture_notes/LectureNotes1.pdf

Update: Long short term memory models are currently doing a great work in predicting the next words. seq2seq models are explained in tensorflow tutorial. There is also a blog post about text generation.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you ! Are you aware of any implementation of this ? (as part of gensim for example). Otherwise, it doesn't seem too complicated to compute. $\endgroup$
    – DED
    Commented Apr 4, 2016 at 15:53
  • $\begingroup$ Hi, could you give me more details about how to retrieve output matrix (syn1 in your example) from trained w2v embedding model? I think the w2v dropped the output matrix when finish training. $\endgroup$
    – Charles Chow
    Commented Aug 16, 2016 at 20:09
  • $\begingroup$ base on my understanding, your answer of 2nd question is to reconstruct the output matrix, is that correct? $\endgroup$
    – Charles Chow
    Commented Aug 16, 2016 at 20:56
  • 1
    $\begingroup$ I think it's worth noting that this is not working like a sequence classifier. The ordering of the words is ignored. $\endgroup$
    – Stefan Falk
    Commented Mar 11, 2017 at 10:32
  • $\begingroup$ One could obtain the output matrix syn1 by just saving the model. yes, the ordering is ignored, for the application one could actually go with LSTM based seq2seq model. $\endgroup$
    – chmodsss
    Commented Aug 7, 2017 at 11:10
6
$\begingroup$

Missing word prediction has been added as a functionality in the latest version of Word2Vec. Of course your sentence need to match the Word2Vec model input syntax used for training the model (lower case letters, stop words, etc)

Usage for predicting the top 3 words for "When I open ? door":

print(model.predict_output_word(['When','I','open','door']), topn = 3)
Improve this answer
$\endgroup$
1
  • $\begingroup$ How did it know that the center was between the 3rd and 4th word? That doesn't make sense to me. I would imagine only even number context words could be put in and it would select the word between floor(len(n)/2)) and floor(len(n)/2))+1 $\endgroup$
    – bmc
    Commented Jul 29, 2019 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.