Why for logistic regression, with target values 0 or 1, it will not work to take the sum of the squares of the difference between target value and prediction, but rather: $$ error({\bf w}) = -1/m * \sum_{i=1}^{m} [ y_i \ln (\sigma({x_i})) + (1-y_i) \ln (1 - \sigma({x_i} ) ] $$


This is the log-likelihood:

$\log P(x; w) \equiv \log \prod_i P(x_i | w) = \sum_i \log P(x_i | w)$, where $P(x_i | w) \equiv \left\{ \begin{array}{rl}\sigma(x_i), & y_i =1 \\ 1 - \sigma(x_i), &y_i = 0\end{array} \right.$

Why the log-likelihood? When you have a probabilistic model, such as logistic regression, it's one way (the MLE) of finding the parameters that fit best. Recall that in logistic regression we are, contrary to the name, trying to classify rather than regress, and the MSE is a regression loss; it seeks to minimize the distance from a point, while we wish to penalize being in the wrong subspace (the parts that don't correspond to the correct class). If you squint a bit, you can see that the negative log-likelihood minimizes the cross entropy.

  • 1
    $\begingroup$ One way of putting the why - if you are 99% certain of a prediction which turns out later to be true, this is 10 times stronger than being 90% certain of an event (think of what you'd be willing to bet), and this is captured by a log-likelihood objective whilst sum of squares difference would prefer you gain from 50% to 60% on some other more marginal example. $\endgroup$ – Neil Slater Apr 16 '16 at 19:07
  • $\begingroup$ @Emre in your notation, the formula you give would be correct even if I used a function different from the logistic function? $\endgroup$ – user Apr 17 '16 at 6:37
  • $\begingroup$ @NeilSlater would you mind expanding a little more your comment? $\endgroup$ – user Apr 17 '16 at 6:39
  • $\begingroup$ Yes, @user, up until the definition of $P(x_i|w)$. $\endgroup$ – Emre Apr 17 '16 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.