Why doesn't neural networks use the concept of degree of freedom?

Question

In most (if not all) NMIST neural network tutorials you will see that the last two layers reduce to a multi-layer perceptron (MLP) and the number of labels is 0-9 for a total of 10 labels. It is well known in statistics that when you have 10 labels, you can set the score of one of them to be 0 and let the other 9 vary. This is the idea of degrees of freedom where 9 variables are allowed to "run free" but one can stay fixed leading to 9 degrees of freedom. More intuitively, when you calculate the probability that an image is a particular number you only need to specify the probabilities of it being one of the 9 numbers, because the probabilities for all ten labels must sum to 1 so the remaining label's probability must be 1 - sum(of the other 9).

The question is: why doesn't neural networks make use of this degree of freedom idea and instead estimate a score for all 10 labels instead of fixing the score of one of them to 0?

Answer 1

Actually neural networks do take this information into account.

When we do a hard classification with NN (i.e. only one class for the output), we (almost always) use a softmax output layer, then taking the max of the softmax layer to get the class and its probability.

Then why do we still have 10 outputs before the softmax and why 10 after? Well, you need to have the 10 inputs to the softmax to indicate the weight the network gives to each class, as this is still not scaled back to a probability. Then to ease the usage of the output layer, we want to keep having 10 outputs to be able to use the values without reconstructing the missing value (which would need to be computed anyway due to the way softmax works).