2
$\begingroup$

I don't understand why one would add additional complexity to log, probabilities for the loss function of a classification Neural Network. What benefit does that have, as opposed to just using the 0-1.0 values(probabilities of a class) you get from the softmax function at the final layer?

Does this add extra non-linearity that we don't understand why it does good, but just happens to do good a lot of times since we give the Neural Net some more complexity?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

Let me explain it with logistic regression first:

  • The output of logistic regression is $\hat{y}= \dfrac{1}{1 + e^{-\theta x}}$
  • Consider the minimum square loss function $\mathcal{L} = (1 -\hat{y})^2$
  • substitute $\hat{y}$ in the second equation $\mathcal{L} = (\dfrac{e^\theta x}{1 + e^{-\theta x}})^2$

Tha above formula is not convex, so optimization is much more difficult since the gradient may point to local optima.

Whereas cross entropy looks like.$$\mathcal{L} = -y\log \hat{y} - (1-y)\log (1-\hat{y}) $$ which has two advantages.

  • It is convex when subtitute $\hat{y}$ for its value (considering that $y$'s value is either 1 or 0)
  • For each row in your dataset you are taking on count both labels at the same time

When we generalize from binari logistic regression to multiclass classification, we use the same ideas of using convex loss functions with respect to the output and also loss functions which minimize at the same time for all labels.

Improve this answer
$\endgroup$
0
$\begingroup$

They are tools for different purposes. Softmax is used in cases that you have labels which are mutually exclusive, they should be contradictory, and exhaustive, one of the labels should always be one while the other is used for cases that there may be multiple labels in the input pattern.

Consider that softmax is just used to face the outputs of a network as probabilities This means that it is a simple function that maps $R^{n}$ space to $R^{n}$ which means softmax has n inputs and n outputs.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Ah, perhaps I didn't ask the question well enough. So yes, that is what softmax does. However, say that softmax gave you .35 for the node that corresponds to the label. Why not just use something like (.35 - 1)^ 2 for the loss function then for that node, and do backprop using that loss function. Why are we instead doing the NLL(negative logistic loss) that looks something more like 1(log(.35)) instead of (.35-1)^2? $\endgroup$
    – katiex7
    Commented Mar 5, 2019 at 5:28
  • $\begingroup$ More importantly, to add to that. Why do we do 1(LOG(.35)), essentially logging our probability here? Couldn't we do something that doesn't involve logging, but a different way to express the loss? Why do we love logging things and why do they work so well? $\endgroup$
    – katiex7
    Commented Mar 5, 2019 at 5:31
  • $\begingroup$ For regression tasks, we use MSE that you've mentioned, but for classification tasks we use logloss and the reason is due to the shape of cost function. The shape of MSE is very bad for classification, it has a very nonlinear noncovex behaviour. $\endgroup$
    – Green Falcon
    Commented Mar 5, 2019 at 5:32
  • $\begingroup$ You can use MSE but that does not have good results. We use the other for fast convergence. $\endgroup$
    – Green Falcon
    Commented Mar 5, 2019 at 5:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.