How can be proved that the softmax output forms a probability distribution and the sigmoid output does not?

Question

I was reading Nielsen's book and in this part of chapter 3 about the softmax function, he says, just before the following Excercise, that the output of a neural network with a output softmax layers forms a probability distribution and the sigmoid output does not always forms it. Now I've been wondering about the output of a neural network, if I have a sigmoid output layer, say for one observation the output is 0.7 for class 0, should the probability for class 1 be 0.3? Or, in this binary classification example, using a softmax output, the first output neuron would be 0.7 for class 0 and 0.3 for class 1 in that particular observation?

Answer 1

Softmax maps $ f:ℝ^n\rightarrow (0,1)^n$ such that $\sum f(\vec x) =1$. Therefore, we can interpret the output of softmax as probabilities.

With sigmoidal activation, there are no such constraints for summation, so even though $ 0<S(\vec x)<1$, it is not guaranteed that $\sum S(\vec x)=1$. The sigmoidal function does not normalize the outputs, so in your example where class 0 has output $0.7$, class 1 could have any value in $(0,1)$, which might not be $0.3$.

---

Here's an example:

$\vec x=[-5,\pi,\frac{1}{3},0] $

$ f(\vec x)\approxeq [2.6379\times10^{-4},0.9059,0.05464]$

$ S(\vec x)\approxeq [6.693\times10^{-3},0.9586,0.5826,0.5] $

Because $0<f(\vec x)<1$ and $\sum f(\vec x)=1$, the softmax output vector can be interpreted as probabilities. On the other hand, $ \sum S(\vec x) > 1$, so you cannot interpret the sigmoidal output as a probability distribution, even though $ 0<S(\vec x)<1$

(I chose the above $\vec x$ arbitrarily to demonstrate that the inputs need not be negative, non-negative, rational, etc., hence $\vec x\in ℝ^n$)