What special characteristics of the softmax function makes it a favourite choice as activation function in output layer of classification neural networks?


The softmax function is simply a generalisation of the logistic function, which simply squashes values into a given range.

At the final layer of a neural network, the model produces its final activations (a.k.a. logits), which we would like to be able to interpret as probabilities, as that would allow is to e.g. create a classification result.

the reason for using the softmax is to ensure these logits all sum up to 1, thereby fulfilling the constraints of a probability density. If we are try to predict whether a medical image contain cancer or not (simply yes or no), then the probability of a positive result (yes) and a negative result (no) must sum up to one. o the model produces a probability vector for each outcome, in pseudo-code: p = [yes, no].

If the final logits in this binary classification example were p = [0.03, 1.92], we can see that they sum to 1.95. This is not interpretable as a probability, although we can see the value is much higher for no. In other examples where there might be 1000s of categories, not just two), we can no-longer assess so easily, which is the largest logit. The softmax gives us some perspective and (quasi-) interpretable probabilities for the categories.


As described by @Neil Slater in his comment below, using the softmax followed by a log loss function does indeed lead to a model that predicts values in the range of true probabilities, thus making them interpretable, as well as providing some nice statistical properties (see the Properties section of Maximum Likelihood Estimation).

Also note: minimising log-loss is equivalent to maximising the Maximium Liklihood Estimation. See a mathematical explanation here.

Implementations of the log-loss include:

These first two method names also highlight that log-loss is also the same as cross-entropy (in the general machine learning context of computing error rates between 0 and 1).

  • $\begingroup$ Thanks for your answer. The answer is clear. Ensure outputs sum up to 1 so that one can easily see which classified choice has the highest probability of being the right one. $\endgroup$ – user781486 Aug 23 '18 at 15:33
  • 1
    $\begingroup$ @user781486 - That is correct. You're welcome! :-) $\endgroup$ – n1k31t4 Aug 23 '18 at 15:34
  • 3
    $\begingroup$ There is more to this. Softmax combined with multi-class logloss should converge close to maximum likelihood estimates. I.e. with the right loss function, the probabilities are not only interpretable, they are statistically valid given the training data. For instance, if you repeated several records with same input $x$ and different classes, the network should learn to output probabilities based on the frequencies of classes seen with that same input. $\endgroup$ – Neil Slater Aug 23 '18 at 15:47
  • $\begingroup$ @NeilSlater - thanks for the input and deeper look. I will edit my answer to reflect your addition. $\endgroup$ – n1k31t4 Aug 23 '18 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.