2
$\begingroup$

I'm reading a TensorFlow tutorial on Word2Vec models and got confused with the objective function. The base softmax function is the following:

$P(w_t|h) = softmax(score(w_t, h) = \frac{exp[score(w_t, h)]}{\Sigma_v exp[score(w',h)]}$, where $score$ computes the compatibility of word $w_t$ with the context $h$ (a dot product is commonly used). We train this model by maximizing its log-likelihood on the training set, i.e. by maximizing $ J_{ML} = \log P(w_t|h) = score(w_t,h) - \log \bigl(\Sigma_v exp[score(w',h)\bigr)$

But why $\log$ disappeared from the $score(w_t,h)$ term?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

No, the logartihm doesn't disappear. From the equation

,

When you want to calculate

, it essentially means calculating ,

Now ,

So ,

as .

Improve this answer
$\endgroup$
2
  • $\begingroup$ So log(exp(x)) log is the natural logarithm I'm I right? $\endgroup$
    – Pauli
    Commented Jan 9, 2021 at 10:20
  • $\begingroup$ Yes, you are correct. That's why log(exp(x)) = x $\endgroup$
    – Gyan Ranjan
    Commented Jan 12, 2021 at 11:20
0
$\begingroup$

It's just an optimisation, for the sake of speed and numerical stability. The two are equivalent for the purpose of determining the gradient since log(x) is monotonically increasing with x.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.