In most (if not all) NMIST neural network tutorials you will see that the last two layers reduce to a multi-layer perceptron (MLP) and the number of labels is 0-9 for a total of 10 labels. It is well known in statistics that when you have 10 labels, you can set the score of one of them to be 0 and let the other 9 vary. This is the idea of degrees of freedom where 9 variables are allowed to "run free" but one can stay fixed leading to 9 degrees of freedom. More intuitively, when you calculate the probability that an image is a particular number you only need to specify the probabilities of it being one of the 9 numbers, because the probabilities for all ten labels must sum to 1 so the remaining label's probability must be 1 - sum(of the other 9)
.
The question is: why doesn't neural networks make use of this degree of freedom idea and instead estimate a score
for all 10 labels instead of fixing the score
of one of them to 0
?