1
$\begingroup$

I was going term by term through the softmax function for the word2vec (SKIP-GRAM) model. I found most definition of these functions to be not 'clear' so I modified the notation to make sure I understand it.

Is the following formulation correct?

$$P(w_{-t} | w_{t} ; \theta) = softmax(score(w_{-t}, w_t))$$

$$P(w_{-t} | w_{t} ; \theta) = \frac{exp(score(w_{-t}, w_t))}{\sum_{w' \in \theta} exp(score(w', w_{t}))}$$

where:

$w_{-t} =$ context

$w_{t} =$ target word

$score(A,B)$ a measure of similarity between vectors A and B.

$\theta = $ vector representation for all words in vocabulary

In the simplest case:

$$score(A ,B) = A \cdot B$$


$\endgroup$
1
$\begingroup$

Your definition is correct. For the reference you can compare it with the probabilistic model from Tensorflow "Vector Representations of Words" tutorial:

$$ \begin{align} P(w_t | h) &= \text{softmax} (\text{score} (w_t, h)) \\ \\ &= \frac{\exp \{ \text{score} (w_t, h) \} } {\sum_\text{Word w' in Vocab} \exp \{ \text{score} (w', h) \}} \end{align} $$

It's the same as yours, but they generalize the condition $h$ as a history.

In case of skip-gram, the target word $w_t$ is any context word ($w_{-t}$ in your notation) and the history is the center word ($w_{t}$ in your notation). You specify $\theta$ explicitly in the condition, but that is usually omitted, because there is only one vocabulary in a given problem.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for this! just one question, in the sum term, does that include scoring w_t to itself? i.e exp(score(w_t, w_t)) $\endgroup$ – SFD Mar 4 '18 at 12:02
  • $\begingroup$ Yes. Just like ordinary softmax. The denominator contains all terms, including the numerator $\endgroup$ – Maxim Mar 4 '18 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.