2
$\begingroup$

I'm reading a TensorFlow tutorial on Word2Vec models and got confused with the objective function. The base softmax function is the following:

$P(w_t|h) = softmax(score(w_t, h) = \frac{exp[score(w_t, h)]}{\Sigma_v exp[score(w',h)]}$, where $score$ computes the compatibility of word $w_t$ with the context $h$ (a dot product is commonly used). We train this model by maximizing its log-likelihood on the training set, i.e. by maximizing $ J_{ML} = \log P(w_t|h) = score(w_t,h) - \log \bigl(\Sigma_v exp[score(w',h)\bigr)$

But why $\log$ disappeared from the $score(w_t,h)$ term?

$\endgroup$
2
$\begingroup$

No , log doesn't disappear . From the equation

,

When you want to calculate

, it essentially means calculating ,

Now ,

So ,

as .

| improve this answer | |
$\endgroup$
0
$\begingroup$

It's just an optimisation, for the sake of speed and numerical stability. The two are equivalent for the purpose of determining the gradient since log(x) is monotonically increasing with x.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.