We know that Softmax usually applied to multi-class labels with the function of $e^{a}\over \sum e^{a}$.
My question is will a function like $a^{2} \over \sum a^{2}$ mostly work also? If not, why?
Here a stand output from last activation.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
In contrast to the alternative you suggested, the advantage of the softmax is that exponentiating works well with log loss, as described in section 6.2.2. of the Deep Learning book:
The aim is to define an activation function which outputs values
[...] between 0 and 1, and [...] the logarithm of the number to be well behaved for gradient-based optimization of the log-likelihood
and,
As with the logistic sigmoid, the use of the exp function works well when training the softmax to output a target value y using maximum log-likelihood. In this case, we wish to maximize $\log P(y=i;z) =\log softmax(z)_i$. Defining the softmax in terms of exp is natural because thelogin the log-likelihood can undo the exp of the softmax [...]
A disadvantage being that
Many objective functions other than the log-likelihood do not work as well with the softmax function. Specifically, objective functions that do not use a log to undo the exp of the softmax fail to learn when the argument to the exp becomes very negative, causing the gradient to vanish. In particular, squared error is a poor loss function for softmax units and can fail to train the model to change its output, even when the model makes highly confident incorrect predictions [...]
For more details I recommend reading the linked section of the book.
