Why use softmax as opposed to standard normalization? In the comment area of the top answer of this question, @Kilian Batzner raised 2 questions which also confuse me a lot. It seems no one gives an explanation except numerical benefits.

I get the reasons for using Cross-Entropy Loss, but how does that relate to the softmax? You said "the softmax function can be seen as trying to minimize the cross-entropy between the predictions and the truth". Suppose, I would use standard / linear normalization, but still use the Cross-Entropy Loss. Then I would also try to minimize the Cross-Entropy. So how is the softmax linked to the Cross-Entropy except for the numerical benefits?

As for the probabilistic view: what is the motivation for looking at log probabilities? The reasoning seems to be a bit like "We use e^x in the softmax, because we interpret x as log-probabilties". With the same reasoning we could say, we use e^e^e^x in the softmax, because we interpret x as log-log-log-probabilities (Exaggerating here, of course). I get the numerical benefits of softmax, but what is the theoretical motivation for using it?

  • $\begingroup$ It is differentiable, leads to non-negative results (such as would be necessary for a probability so the cross-entropy can be calculated), and behaves like the max function, which is appropriate in a classification setting. Welcome to the site! $\endgroup$ – Emre Sep 20 '17 at 6:09
  • $\begingroup$ @Emre Thanks! But what does "behaves like max function" mean? Besides, if I have another function that is also differentiable, monotone increasing and leads to non-negative results, can I use it to replace the exp function in the formula? $\endgroup$ – Hans Sep 20 '17 at 7:12
  • $\begingroup$ When you normalize using $\max$, the greatest argument gets mapped to 1 while the rest get mapped to zero, owing to the growth of the exponential fuction. $\endgroup$ – Emre Sep 20 '17 at 16:02

It is more than just numerical. A quick reminder of the softmax: $$ P(y=j | x) = \frac{e^{x_j}}{\sum_{k=1}^K e^{x_k}} $$

Where $x$ is an input vector with length equal to the number of classes $K$. The softmax function has 3 very nice properties: 1. it normalizes your data (outputs a proper probability distribution), 2. is differentiable, and 3. it uses the exp you mentioned. A few important points:

  1. The loss function is not directly related to softmax. You can use standard normalization and still use cross-entropy.

  2. A "hardmax" function (i.e. argmax) is not differentiable. The softmax gives at least a minimal amount of probability to all elements in the output vector, and so is nicely differentiable, hence the term "soft" in softmax.

  3. Now I get to your question. The $e$ in softmax is the natural exponential function. Before we normalize, we transform $x$ as in the graph of $e^x$:

natural exponential function

If $x$ is 0 then $y=1$, if $x$ is 1, then $y=2.7$, and if $x$ is 2, now $y=7$! A huge step! This is what's called a non-linear transformation of our unnormalized log scores. The interesting property of the exponential function combined with the normalization in the softmax is that high scores in $x$ become much more probable than low scores.

An example. Say $K=4$, and your log score $x$ is vector $[2, 4, 2, 1]$. The simple argmax function outputs:

$$ [0, 1, 0, 0] $$

The argmax is the goal, but it's not differentiable and we can't train our model with it :( A simple normalization, which is differentiable, outputs the following probabilities:

$$ [0.2222, 0.4444, 0.2222, 0.1111] $$

That's really far from the argmax! :( Whereas the softmax outputs: $$ [0.1025, 0.7573, 0.1025, 0.0377] $$

That's much closer to the argmax! Because we use the natural exponential, we hugely increase the probability of the biggest score and decrease the probability of the lower scores when compared with standard normalization. Hence the "max" in softmax.

  • $\begingroup$ @Hans, awesome! If this answered your question, please consider clicking it as answered $\endgroup$ – vega Oct 31 '17 at 18:59
  • 3
    $\begingroup$ Great info. However, instead of using e, what about using a constant say 3, or 4 ? Will the outcome be the same? $\endgroup$ – Cheok Yan Cheng Nov 23 '17 at 20:45
  • 4
    $\begingroup$ @CheokYanCheng, yes. But e has a nicer derivative ;) $\endgroup$ – vega Nov 25 '17 at 23:55
  • $\begingroup$ I have seen that the result of softmax is typically used as the probabilities of belonging to each class. If the choice of 'e' instead of other constant is arbitrary, it does not make sense to see it in terms of probability, right? $\endgroup$ – jvalle Oct 26 '18 at 21:13
  • $\begingroup$ @vega Sorry, but I still don't see how that answers the question: why not use e^e^e^e^e^x for the very same reasons? Please explain $\endgroup$ – Gulzar Dec 9 '18 at 15:39

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.