4
$\begingroup$

On page 12, we take $log$ on both side.

$\max_{\boldsymbol{w}}L\boldsymbol({w})=\max_{w}\displaystyle\prod_{n=1}^Np(t^{(i)}|x^{(i)};\boldsymbol{w})$

$\ell(\boldsymbol{w})=-logL(\boldsymbol{w})$

$\ \ \ \ \ \ \ =-\displaystyle\sum_{i=1}^Nlog\ p(t^{(i)}|x^{(i)};\boldsymbol{w})$

The $log$ function is increasing as $\boldsymbol{w}$ increase. Why we have to take the negative?

$\endgroup$

1 Answer 1

4
$\begingroup$

It is common to define optimization problems as minimization problems instead of maximization. And by multiplying your target functions with $-1$ you can transform one into the other:

$$\max_{w} \log{L(w)} \Leftrightarrow \min_{w} -log{L(w)}$$

So to maximize the log-likelihood you minimize the negative log-likelihood. Basically it just comes down to conventions in optimization theory.

Moreover, since $L(w) \in [0,1]$ its logarithm $\log{L(w)}$ will be less than or equal to $0$ (note that $log{0}$ is not defined). Accordingly $\max_{w} \log{L(w)}$ means to maximize a negative number which is, at least to me, less intuitive than minimizing a positive number.

The more interesting part is actually the log-transformation which increases numerical stability of your calculations (since it "transforms" the multiplication to a sum and thereby reduces the risk of underflowing).

$\endgroup$
4
  • $\begingroup$ can you please tell me why log is needed at first hand ? can you please explain this line more with an example maxwlogL(w) means to maximize a negative number ? why we need to maximize when th intention is to minimize the loss? $\endgroup$ Jan 3, 2020 at 14:22
  • 1
    $\begingroup$ @Aj_MLstater $\max_{\boldsymbol{w}}L\boldsymbol({w})=\max_{w}\displaystyle\prod_{n=1}^Np(t^{(i)}|x^{(i)};\boldsymbol{w})$ is a product. The problem with this product is that it consists of probabilities which are per definition between $0$ and $1$. And a product of many number between $0$ and $1$ is very small which leads to numerical problems for computers. However, taking the $\log$ of it gives you a sum since $\log{ab} = \log{a} + \log{b}$. And that is easier to handle for computers. $\endgroup$
    – Sammy
    Jan 4, 2020 at 9:43
  • $\begingroup$ thanks for the infromation ,will the result of log of any value will between 0 and 1 ? just like how probablitiy of any value is between 0 and 1 . am i right ? $\endgroup$ Jan 4, 2020 at 20:45
  • $\begingroup$ @Aj_MLstater Not the logarithm but $p(t^{(i)}|x^{(i)};\boldsymbol{w})$ is between $0$ and $1$. I suggest to read en.wikipedia.org/wiki/Logarithm for more information. Looking at the graphs on the right hand side provides a quick understanding of how the $log$ behaves for different bases and input values. $\endgroup$
    – Sammy
    Jan 5, 2020 at 7:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.