5
$\begingroup$

In most (if not all) NMIST neural network tutorials you will see that the last two layers reduce to a multi-layer perceptron (MLP) and the number of labels is 0-9 for a total of 10 labels. It is well known in statistics that when you have 10 labels, you can set the score of one of them to be 0 and let the other 9 vary. This is the idea of degrees of freedom where 9 variables are allowed to "run free" but one can stay fixed leading to 9 degrees of freedom. More intuitively, when you calculate the probability that an image is a particular number you only need to specify the probabilities of it being one of the 9 numbers, because the probabilities for all ten labels must sum to 1 so the remaining label's probability must be 1 - sum(of the other 9).

The question is: why doesn't neural networks make use of this degree of freedom idea and instead estimate a score for all 10 labels instead of fixing the score of one of them to 0?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Actually neural networks do take this information into account.

When we do a hard classification with NN (i.e. only one class for the output), we (almost always) use a softmax output layer, then taking the max of the softmax layer to get the class and its probability.

Then why do we still have 10 outputs before the softmax and why 10 after? Well, you need to have the 10 inputs to the softmax to indicate the weight the network gives to each class, as this is still not scaled back to a probability. Then to ease the usage of the output layer, we want to keep having 10 outputs to be able to use the values without reconstructing the missing value (which would need to be computed anyway due to the way softmax works).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Sorry this sentence didn't make sense to me - "we want to keep having 10 outputs to be able to use the values without reconstructing the missing value". Can you please explain further? With an example perhaps. $\endgroup$
    – xiaodai
    Jan 6, 2019 at 21:52
  • $\begingroup$ Well, we don't want to have to reconstruct the missing value. It's easier when doing inference to have all the values. And it has to be computed anyway, so better to make it available! $\endgroup$
    – Matthieu Brucher
    Jan 6, 2019 at 23:37
  • $\begingroup$ The "missing value" is just 0. So I don't understand the point. It's pretty simple, so I don't why make it available as something other than 0. $\endgroup$
    – xiaodai
    Jan 7, 2019 at 3:04
  • $\begingroup$ It's a probability, not just one value. $\endgroup$
    – Matthieu Brucher
    Jan 7, 2019 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.