Why take softmax at all at the final layer for multi-class classification problems? For example softmax of the vector [1, .5]

Is [.621, .379]

I mean if we just took the straight ratio, it'd give me [.667, .333] instead

Does that really make a difference?

Is it cause the vector can have negative numbers that we softmax things? What benefit do we get from making an odder way to give a ratio/probability to certain numbers as opposed to just taking a ratio of the numbers?

  • $\begingroup$ Softmax is exactly that ratio, but the differentiable one. $\endgroup$ – Media Mar 5 '19 at 5:50
  • $\begingroup$ Could I ask for more elaboration? From what I know, the ratio function is also differentiable $\endgroup$ – katiex7 Mar 5 '19 at 5:52
  • $\begingroup$ For example you can find the derivative of xi/summation(xvec) by using the quotient rule $\endgroup$ – katiex7 Mar 5 '19 at 5:53
  • 1
    $\begingroup$ Yes, you are finding softmax not ratio. $\endgroup$ – Media Mar 5 '19 at 5:57
  • 1
    $\begingroup$ Oops, you are referring to something else that I didn't notice. The point is that in the last layer you may have tanh activation function which may lead to negative outcomes. This will affect the simple sum which is going to be in the denominator. Consequently, we use exponentiation to avoid wrong ratios. Due to the fact that they should be valid probabilities. $\endgroup$ – Media Mar 5 '19 at 6:00

The ratio doesn't take into account the fact that the last layer may have negatives results, in which case the ratio doesn't work, while the softmax does. Another corner case is if the denominator of the ratio is zero.

Also, softmax has two interesting properties:

  1. It will accentuate the differences between the input and the output, by squashing the lower values and increasing the highest value, which makes the output of the network closert to a max instead of a softmax (note that max is not differentiable and hence cannot be used).
  2. In a classification problem, where a softmax layer is used, the loss function is the cross entropy, which has the form - y * log(y_hat). Turns out that log of softmax is very easy to compute, as log(a/b) is log(a) - log(b) and log(exp(x)) is simply x. That makes softmax more desired.
| improve this answer | |
  • $\begingroup$ Everything is good, but i have a question. Why is max not differentiable? $\endgroup$ – katiex7 Mar 5 '19 at 15:33
  • 1
    $\begingroup$ With an example: max([4,3,2]) will be 4. That operation can be understand as: multiply by 1 the element where there is the maximum (4) otherwise by zero, and sum the result. Implicitly one is multiplying by the step function, which is not differentiable. $\endgroup$ – Escachator Mar 6 '19 at 17:43
  • $\begingroup$ Excellent, i love upvoting posts like these. Btw rereading your post again on "Turns out that log of softmax is very easy to compute, as log(a/b) is log(a) - log(b) and log(exp(x)) is simply x. That makes softmax more desired." Did you simply mean to say its fast to compute this? If so, can you apply the same division becoming subtraction for log(exp(x1)/(exp(x1) +exp(x2))? Which is just a softmax of two classes $\endgroup$ – katiex7 Mar 7 '19 at 2:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.