I am trying to understand the loss function which is used for the word2vec model, but I don't really follow the argumentation behind this video https://www.youtube.com/watch?v=ERibwqs9p38&t=5s, at 29:30.

The formula which is unclear is the following:

$J(\theta) = \displaystyle-\dfrac{1}{T} {\sum_{t=1}^{T} \sum_{-m <= j <=m, \\j\ne0} log(p(w_{t+j}|w_t))}.$

  • $T$ is the number of words in the vocabulary
  • $w_t$ is a given word and we try to calculate the probablity that another word $w_{t+j}$ occurs within a window of +/- $m$ words ahead.
  • $\theta$ is the solution we're after. It is essentially a $2*d*Tx1$ dimensional vector which contains all the columns of the matrices V and U.

At a first sight it looks all pretty clear: we're iterating though the whole vocabulary and for each (fixed word then), we add up all probabilities that another word occurs within a window around that fixed word.

However, it fails apart for me when I consider that a word $w_t$ occurs in many positions in the corpus and might occur multiple times with a different word. E.g, the words 'deep learning' often occur together, which indicates that there's a contextual relation between them. Why would we only count them twice? It seems like the formula above is counting each pair $p(w_{t+j}|w_t)$ just twice (e.g. once for $p(deep|learning)$ and once for $ p(learning|deep)$). IMO we should need a correcting term that adjusts for the missing 'frequency', e.g.

$J(\theta) = \displaystyle-\dfrac{1}{T} {\sum_{t=1}^{T} \sum_{-m <= j <=m, \\j\ne0} log(\lambda(w_t,w_{t+j})p(w_{t+j}|w_t))}.$

In the case where $\lambda(x,y) = 1$, we get the formula above, but we could also be free to chose a function that boosts frequent occurrences (pairwise). The formula above could then be seen as a special case when you don't care that words that occur more often together get a boost.

On the other hand, when the formula above already accounts for multiple occurrences, then where is this visible? The author then continues and defines $p(o|w)$ as $exp(o^T*w)/\sum(exp(u^T*w))$. In this particular case, I don't see that we're counting the dot-product as many times as the word $o$ is in the neighbourhood of the word $w$. Maybe the choice of $p$ is just very simplistic and represents a model where the fact that 2 words are in the neighbourhood (just somewhere in the corpus) is enough (bag of words model???). It's hard to see then that such models deliver good performance in NLP though.


1 Answer 1


Not sure what the video said, but $T$ should not be the vocabulary size, but the training corpus size (number of all words).

For example, if your training corpus is

deep learning is popular . i love deep learning . i want to learn more about it.

Then when you sume up over $T$, you will sum up all the word pairs in the corpus including duplicates. The word pair (deep learning) is indeed calculated twice.

For details please refer to the original skip-gram paper, and notice the definitoin (1) on page 2.

  • $\begingroup$ So if T is the corpus size and not the vocabulary size, then this changes everything. It was not clear from the formula what T was. Thank you for the clarification $\endgroup$
    – shaft
    Commented Sep 21, 2018 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.