# Why do we Softmax at all?

Why take softmax at all at the final layer for multi-class classification problems? For example softmax of the vector [1, .5]

Is [.621, .379]

I mean if we just took the straight ratio, it'd give me [.667, .333] instead

Does that really make a difference?

Is it cause the vector can have negative numbers that we softmax things? What benefit do we get from making an odder way to give a ratio/probability to certain numbers as opposed to just taking a ratio of the numbers?

• Softmax is exactly that ratio, but the differentiable one. Commented Mar 5, 2019 at 5:50
• Could I ask for more elaboration? From what I know, the ratio function is also differentiable Commented Mar 5, 2019 at 5:52
• For example you can find the derivative of xi/summation(xvec) by using the quotient rule Commented Mar 5, 2019 at 5:53
• Yes, you are finding softmax not ratio. Commented Mar 5, 2019 at 5:57
• Oops, you are referring to something else that I didn't notice. The point is that in the last layer you may have tanh activation function which may lead to negative outcomes. This will affect the simple sum which is going to be in the denominator. Consequently, we use exponentiation to avoid wrong ratios. Due to the fact that they should be valid probabilities. Commented Mar 5, 2019 at 6:00

## 1 Answer

The ratio doesn't take into account the fact that the last layer may have negatives results, in which case the ratio doesn't work, while the softmax does. Another corner case is if the denominator of the ratio is zero.

Also, softmax has two interesting properties:

1. It will accentuate the differences between the input and the output, by squashing the lower values and increasing the highest value, which makes the output of the network closert to a max instead of a softmax (note that max is not differentiable and hence cannot be used).
2. In a classification problem, where a softmax layer is used, the loss function is the cross entropy, which has the form - y * log(y_hat). Turns out that log of softmax is very easy to compute, as log(a/b) is log(a) - log(b) and log(exp(x)) is simply x. That makes softmax more desired.
• Everything is good, but i have a question. Why is max not differentiable? Commented Mar 5, 2019 at 15:33
• With an example: max([4,3,2]) will be 4. That operation can be understand as: multiply by 1 the element where there is the maximum (4) otherwise by zero, and sum the result. Implicitly one is multiplying by the step function, which is not differentiable. Commented Mar 6, 2019 at 17:43
• Excellent, i love upvoting posts like these. Btw rereading your post again on "Turns out that log of softmax is very easy to compute, as log(a/b) is log(a) - log(b) and log(exp(x)) is simply x. That makes softmax more desired." Did you simply mean to say its fast to compute this? If so, can you apply the same division becoming subtraction for log(exp(x1)/(exp(x1) +exp(x2))? Which is just a softmax of two classes Commented Mar 7, 2019 at 2:28