We know that Softmax usually applied to multi-class labels with the function of $e^{a}\over \sum e^{a}$.

My question is will a function like $a^{2} \over \sum a^{2}$ mostly work also? If not, why?

Here a stand output from last activation.


1 Answer 1


In contrast to the alternative you suggested, the advantage of the softmax is that exponentiating works well with log loss, as described in section 6.2.2. of the Deep Learning book:

The aim is to define an activation function which outputs values

[...] between 0 and 1, and [...] the logarithm of the number to be well behaved for gradient-based optimization of the log-likelihood


As with the logistic sigmoid, the use of the exp function works well when training the softmax to output a target value y using maximum log-likelihood. In this case, we wish to maximize $\log P(y=i;z) =\log softmax(z)_i$. Defining the softmax in terms of exp is natural because thelogin the log-likelihood can undo the exp of the softmax [...]

A disadvantage being that

Many objective functions other than the log-likelihood do not work as well with the softmax function. Specifically, objective functions that do not use a log to undo the exp of the softmax fail to learn when the argument to the exp becomes very negative, causing the gradient to vanish. In particular, squared error is a poor loss function for softmax units and can fail to train the model to change its output, even when the model makes highly confident incorrect predictions [...]

For more details I recommend reading the linked section of the book.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.