I'm reading a TensorFlow tutorial on Word2Vec models and got confused with the objective function. The base softmax function is the following:

$$P(w_t|h) = softmax(score(w_t, h) = \frac{exp[score(w_t, h)]}{\Sigma_v exp[score(w',h)]}$$, where $$score$$ computes the compatibility of word $$w_t$$ with the context $$h$$ (a dot product is commonly used). We train this model by maximizing its log-likelihood on the training set, i.e. by maximizing $$J_{ML} = \log P(w_t|h) = score(w_t,h) - \log \bigl(\Sigma_v exp[score(w',h)\bigr)$$

But why $$\log$$ disappeared from the $$score(w_t,h)$$ term?

No, the logartihm doesn't disappear. From the equation

$P(w_t|h)&space;=&space;\frac{exp[score(w_t,&space;h)]}{\Sigma_v&space;exp[score(w',h)]}$ ,

When you want to calculate

$log(P(w_t|h)$ , it essentially means calculating , $log&space;\frac{exp[score(w_t,&space;h)]}{\Sigma_v&space;exp[score(w',h)]}$

Now , $log&space;\frac&space;AB&space;=&space;log&space;A&space;-&space;log&space;B$

So ,

$log&space;\frac{exp[score(w_t,&space;h)]}{\Sigma_v&space;exp[score(w',h)]}&space;=&space;log&space;(exp[score(w_t,&space;h)])&space;-&space;log(\Sigma_v&space;exp[score(w',h)])$

$=score(w_t,&space;h)&space;-&space;log(\Sigma_v&space;exp[score(w',h)])$

as $log(exp(x))&space;=&space;x$.

• So log(exp(x)) log is the natural logarithm I'm I right? – Pauli Jan 9 at 10:20
• Yes, you are correct. That's why log(exp(x)) = x – Gyan Ranjan Jan 12 at 11:20

It's just an optimisation, for the sake of speed and numerical stability. The two are equivalent for the purpose of determining the gradient since log(x) is monotonically increasing with x.