I am reading Michael Nielsen's book on deep learning. In the first chapter, he gives the classic example of classifying 10 handwritten digits, and uses it to explain the intuition behind choosing the number of output neurons.

Initially, before reading the book, I thought it was intuitive that you would want 10 output neurons, each representing a given digit, but I'm starting to question why I thought so.

Anyways, here's what Nielsen explains:

You might wonder why we use 10 output neurons. After all, the goal of the network is to tell us which digit (0,1,2,…,9) corresponds to the input image. A seemingly natural way of doing that is to use just 4 output neurons, treating each neuron as taking on a binary value, depending on whether the neuron's output is closer to 0 or to 1. Four neurons are enough to encode the answer, since 2^4=16 is more than the 10 possible values for the input digit. Why should our network use 10 neurons instead? Isn't that inefficient? The ultimate justification is empirical: we can try out both network designs, and it turns out that, for this particular problem, the network with 10 output neurons learns to recognize digits better than the network with 4 output neurons. But that leaves us wondering why using 10 output neurons works better. Is there some heuristic that would tell us in advance that we should use the 10-output encoding instead of the 4 -output encoding? To understand why we do this, it helps to think about what the neural network is doing from first principles. Consider first the case where we use 10 output neurons. Let's concentrate on the first output neuron, the one that's trying to decide whether or not the digit is a 0. It does this by weighing up evidence from the hidden layer of neurons. What are those hidden neurons doing? Well, just suppose for the sake of argument that the first neuron in the hidden layer detects whether or not an image like the following is present:

enter image description here

It can do this by heavily weighting input pixels which overlap with the image, and only lightly weighting the other inputs. In a similar way, let's suppose for the sake of argument that the second, third, and fourth neurons in the hidden layer detect whether or not the following images are present:

enter image description here

As you may have guessed, these four images together make up the 0 image that we saw in the line of digits shown earlier:

enter image description here

So if all four of these hidden neurons are firing then we can conclude that the digit is a 0. Of course, that's not the only sort of evidence we can use to conclude that the image was a 0 - we could legitimately get a 0 in many other ways (say, through translations of the above images, or slight distortions). But it seems safe to say that at least in this case we'd conclude that the input was a 0.

Supposing the neural network functions in this way, we can give a plausible explanation for why it's better to have 10 outputs from the network, rather than 4. If we had 4 outputs, then the first output neuron would be trying to decide what the most significant bit of the digit was. And there's no easy way to relate that most significant bit to simple shapes like those shown above. It's hard to imagine that there's any good historical reason the component shapes of the digit will be closely related to (say) the most significant bit in the output.

Now, with all that said, this is all just a heuristic. Nothing says that the three-layer neural network has to operate in the way I described, with the hidden neurons detecting simple component shapes. Maybe a clever learning algorithm will find some assignment of weights that lets us use only 4 output neurons. But as a heuristic the way of thinking I've described works pretty well, and can save you a lot of time in designing good neural network architectures.

I don't understand what he means in the second to last paragraph. I included the rest for context, but it's this second to last paragraph that's confusing me. Could someone clarify what he's talking about?


The reason 10 neurons work better than 4 is because it allows the network to encode all possible answers independently.

You could have 4 neurons and train the first one to encode the least significant bit, the second neuron the second-least significant bit, and so on. It would work.

Images of a 1 (bits: 0001) are quite like images of a 7 (bits 0111). Imagine an image where we know it’s a 1 or a 7, but we have no idea which one. With 4 outputs (as described) you would output (0, 0.5, 0.5, 1), meaning the correct answer is either 1, 3, 5, or 7. With 10 outputs, all classes would have 0 probability and 1 and 7 would have 50% each, which conveys more accurately what can be interpreted from the image.

You could have a single output that just outputs the value of the digit directly. But a 1 and 7 are now far apart in the output node (the whole range is from 0 to 9), so it’s impossible for this network to say “1 or 7”. It will probably say: 4 (midpoint between 1 and 7). That’s why it doesn’t work well.

Very poorly explained by “Nielsen’s book”, IMHO.


Michael Nielsen is discussing the most significant bit of a digit for binary representation of numbers.

That section can be ignored neural networks do not learn the binary, aka base-2, representation of MNIST numbers. Neural networks learn the visual representations of decimal, aka base-10, numbers.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.