I was reading Nielsen's book and in this part of chapter 3 about the softmax function, he says, just before the following Excercise, that the output of a neural network with a output softmax layers forms a probability distribution and the sigmoid output does not always forms it. Now I've been wondering about the output of a neural network, if I have a sigmoid output layer, say for one observation the output is 0.7 for class 0, should the probability for class 1 be 0.3? Or, in this binary classification example, using a softmax output, the first output neuron would be 0.7 for class 0 and 0.3 for class 1 in that particular observation?
1 Answer
Softmax maps $ f:ℝ^n\rightarrow (0,1)^n$ such that $\sum f(\vec x) =1$. Therefore, we can interpret the output of softmax as probabilities.
With sigmoidal activation, there are no such constraints for summation, so even though $ 0<S(\vec x)<1$, it is not guaranteed that $\sum S(\vec x)=1$. The sigmoidal function does not normalize the outputs, so in your example where class 0
has output $0.7$, class 1
could have any value in $(0,1)$, which might not be $0.3$.
Here's an example:
$\vec x=[-5,\pi,\frac{1}{3},0] $
$ f(\vec x)\approxeq [2.6379\times10^{-4},0.9059,0.05464]$
$ S(\vec x)\approxeq [6.693\times10^{-3},0.9586,0.5826,0.5] $
Because $0<f(\vec x)<1$ and $\sum f(\vec x)=1$, the softmax output vector can be interpreted as probabilities. On the other hand, $ \sum S(\vec x) > 1$, so you cannot interpret the sigmoidal output as a probability distribution, even though $ 0<S(\vec x)<1$
(I chose the above $\vec x$ arbitrarily to demonstrate that the inputs need not be negative, non-negative, rational, etc., hence $\vec x\in ℝ^n$)
-
$\begingroup$ Thank you! So, in your example, I can see that the sum of all f(x) is 1 for the observation x, but for S(x), why do you sum it when actually, by using sigmoidal output, isn't it just one number (eg, 0.7 of being 1) because in that case we only use one output sigmoid neuron? Being that the case of one output neuron, wouldn't the probability of being 0 be 0.3? as in, for example, logistic regression? All these is like the second part of the question I coulnd't get to ask. $\endgroup$ Aug 10, 2020 at 3:57
-
$\begingroup$ For binary classification, a single sigmoidal output neuron would be sufficient for the reasons you stated. For multiclass classification, sigmoidal neurons aren't guaranteed (and probably won't) output a legitimate probability distribution for the reasons I stated. So yes, in a binary classification problem, a single sigmoidal neuron is totally fine. $\endgroup$– BenAug 10, 2020 at 15:32
-
$\begingroup$ Thank you very much, sir! Would you recommend any books or resources to deepen the understanding of this topic? I am finishing Nielsen's book, and then go through Stanford's NLP and CNN courses. $\endgroup$ Aug 10, 2020 at 18:18
-
$\begingroup$ Unfortunately, I am not up-to-date with reading materials for fundamentals. However, I'd recommend reading highly cited papers after you understand the basic concepts. Good luck! $\endgroup$– BenAug 10, 2020 at 18:40