# Why do we Softmax at all?

Why take softmax at all at the final layer for multi-class classification problems? For example softmax of the vector [1, .5]

Is [.621, .379]

I mean if we just took the straight ratio, it'd give me [.667, .333] instead

Does that really make a difference?

Is it cause the vector can have negative numbers that we softmax things? What benefit do we get from making an odder way to give a ratio/probability to certain numbers as opposed to just taking a ratio of the numbers?

• Softmax is exactly that ratio, but the differentiable one. Mar 5 '19 at 5:50
• Could I ask for more elaboration? From what I know, the ratio function is also differentiable Mar 5 '19 at 5:52
• For example you can find the derivative of xi/summation(xvec) by using the quotient rule Mar 5 '19 at 5:53
• Yes, you are finding softmax not ratio. Mar 5 '19 at 5:57
• Oops, you are referring to something else that I didn't notice. The point is that in the last layer you may have tanh activation function which may lead to negative outcomes. This will affect the simple sum which is going to be in the denominator. Consequently, we use exponentiation to avoid wrong ratios. Due to the fact that they should be valid probabilities. Mar 5 '19 at 6:00

The ratio doesn't take into account the fact that the last layer may have negatives results, in which case the ratio doesn't work, while the softmax does. Another corner case is if the denominator of the ratio is zero.
1. It will accentuate the differences between the input and the output, by squashing the lower values and increasing the highest value, which makes the output of the network closert to a max instead of a softmax (note that max is not differentiable and hence cannot be used).
2. In a classification problem, where a softmax layer is used, the loss function is the cross entropy, which has the form - y * log(y_hat). Turns out that log of softmax is very easy to compute, as log(a/b) is log(a) - log(b) and log(exp(x)) is simply x. That makes softmax more desired.
• With an example: max([4,3,2]) will be 4. That operation can be understand as: multiply by 1 the element where there is the maximum (4) otherwise by zero, and sum the result. Implicitly one is multiplying by the step function, which is not differentiable. Mar 6 '19 at 17:43