Understanding softmax layer in Deep Neural Network

In a DNN, once the logits vector is produced say

$$[y_0,y_1]$$ where the number of neuron in the logits layer is 2, the condition holds where $$y_0 >= 0$$ and $$y_1<= 1$$.

This vector is then passed into the softmax layer where it squeezes the value to be within the range of 0 to 1. However, how do i prove that $$y_0$$ will always be $$>=0$$ and $$y_1$$ will always be $$<=1$$?

From the way i understand, both $$y_0 + y_1 = 1$$ hence $$y_0 >= 0$$ and $$y_1 <=1$$ but i find that it doesn't really prove on why $$y_0 >= 0$$ and $$y_1 <=1$$. Could anyone assist me on my understanding.

Look at the definition of softmax: https://en.wikipedia.org/wiki/Softmax_function

The exp() gives a positive value. Since you "standardize" each value exp(e^z) in the numerator by the sum of all exp(e^z), the values sum to one in total.

Is whis what you are looking for?

Maybe you should have a look at Hasties et al. "Elements of Statistical Learning" (Ch. 11.2 "Neural Nets").

Little R example:

# For standardization (denominator)
base = exp(-2) + exp(5)

# Single exp() values are positive
exp(-2)
[1] 0.1353353
exp(5)
[1] 148.4132

# Standatdize
exp(-2)/base
[1] 0.0009110512
exp(5)/base
[1] 0.9990889

# Total sum equals 1
exp(-2)/base + exp(5)/base
[1] 1

• thank you. So instead of negative, if i provide a value more than 1, say exp(5) /base the value would be standardized to {0,1}? So the exp() handles values that are negative and positive, squeezing their range to be between 0 and 1? May 22, 2019 at 9:50
• 1) The exponential function exp() makes values z "non-negative". 2) Dividing each single exp(z) by the sum of all exp(z) ensures that the exp(z) in total equals one. Consequently, a single sigma(z) must be >=0 and <= 1. If you feel that this helped you, a vote is apprechiated. en.wikipedia.org/wiki/Exponential_function May 22, 2019 at 10:06
• thank you so much. Appreciated it. Fully understood! Cheers May 22, 2019 at 10:07