0
$\begingroup$

From this article:

  • In vanilla skip gram model, softmax is computationally very expensive, as it requires scanning through the entire output embedding matrix (W_output) to compute the probability distribution of all V words, where V can be millions or more.
  • Furtheremore, the normalization factor in the denominator also requires V iterations.

Hence the article suggests applying negative sampling instead of softmax. There are many article discussing skip grams with negative sampling. But I did not find any discussing CBOW (Continuous Bag of Words Model ) model with negative sampling. Why is this so? Is it not possible / recommended? Or its exacty same as skip gram? Can you please shed some insights about using negative sampling with CBOW?

PS: any article / paper link will also be of great help.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Just from the perspective of finding relevant article[s] this one seems aligned: NLP’s word2vec: Negative Sampling Explained . The article seems to address the topics you mention. In particular : skip-grams, computational questions, negative sampling, and the objective function to optimize.

As far as being able to follow it there are gaps in its explanations: so it is a bit of gymnastics to read it.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thats specific to negative sampling in the context of skip gram and have come across it earlier. I am looking for details of using negative sampling in CBOW. I believe there wont be much difference. I guess I understand negative sampling quite well, at least in the context of skip grams. I guess I should try it myself, implementation of negative sampling in CBOW. May be I succeed or I fail and come up with fresh doubts... $\endgroup$
    – Mahesha999
    Feb 14 at 12:30
  • $\begingroup$ OK let us know what you find out! $\endgroup$
    – WestCoastProjects
    Feb 14 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.