# Are deep learning models way over the required capacity for their datasets' estimated entropies?

this question might seem a bit odd. I was doing some self-studies into information theory and decided to do some more formal investigations into deep learning. Please bear with me as I try to explain. I took a large "training" subset of MNIST as my guinea pig.

1) Converted every image in MNIST into "black-and-white" (pixels values only 0 or 1)

2) Summed over all data images to build a histogram over the pixels - I counted the number of times each pixel gets a 1 value in the dataset

3) Normalized histogram to get an estimate of the "true" probability distribution

4) From this, I got the following probability distribution (shown as a heatmap with matplotlib):

[ 5) Now I calculated the entropy and got: $191$ bits

6) According to David MacKay in his Information Theory book, we could interpret a neural network as a noisy channel and consider each neuron as having a 2 bit capacity. Although he does state to use this idea with care. Chapter 40 of his book http://www.inference.org.uk/itila/book.html)

7) So, as a rough estimate (and with care) we could say we would need a neural network of 95 neurons in order to be able to encode the labeling of this MNIST training set (190/2) 8) Now we can get to my question:

Even if this is a very "back-of-the-envelope" calculation, shouldn't a neural network capable of learning the labelling be at least in the ballpark of 95 neurons? Why do we need, for instance, a neural network with 21840 parameters to get 99% accuracy? (considering the one in PyTorch's example for MNIST: https://github.com/pytorch/examples/blob/master/mnist/main.py)