0
$\begingroup$

In a DNN, once the logits vector is produced say

$[y_0,y_1]$ where the number of neuron in the logits layer is 2, the condition holds where $y_0 >= 0$ and $y_1<= 1$.

This vector is then passed into the softmax layer where it squeezes the value to be within the range of 0 to 1. However, how do i prove that $y_0$ will always be $>=0$ and $y_1$ will always be $<=1$?

From the way i understand, both $y_0 + y_1 = 1$ hence $y_0 >= 0$ and $y_1 <=1$ but i find that it doesn't really prove on why $y_0 >= 0$ and $y_1 <=1$. Could anyone assist me on my understanding.

$\endgroup$

1 Answer 1

0
$\begingroup$

Look at the definition of softmax: https://en.wikipedia.org/wiki/Softmax_function

enter image description here

The exp() gives a positive value. Since you "standardize" each value exp(e^z) in the numerator by the sum of all exp(e^z), the values sum to one in total.

Is whis what you are looking for?

Maybe you should have a look at Hasties et al. "Elements of Statistical Learning" (Ch. 11.2 "Neural Nets").

Little R example:

# For standardization (denominator)
base = exp(-2) + exp(5) 

# Single exp() values are positive
exp(-2)
[1] 0.1353353
exp(5)
[1] 148.4132

# Standatdize
exp(-2)/base 
[1] 0.0009110512
exp(5)/base 
[1] 0.9990889

# Total sum equals 1
exp(-2)/base + exp(5)/base 
[1] 1
$\endgroup$
3
  • $\begingroup$ thank you. So instead of negative, if i provide a value more than 1, say exp(5) /base the value would be standardized to {0,1}? So the exp() handles values that are negative and positive, squeezing their range to be between 0 and 1? $\endgroup$
    – Maxxx
    May 22, 2019 at 9:50
  • $\begingroup$ 1) The exponential function exp() makes values z "non-negative". 2) Dividing each single exp(z) by the sum of all exp(z) ensures that the exp(z) in total equals one. Consequently, a single sigma(z) must be >=0 and <= 1. If you feel that this helped you, a vote is apprechiated. en.wikipedia.org/wiki/Exponential_function $\endgroup$
    – Peter
    May 22, 2019 at 10:06
  • $\begingroup$ thank you so much. Appreciated it. Fully understood! Cheers $\endgroup$
    – Maxxx
    May 22, 2019 at 10:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.