why is more layers better in deep learning?

Question

i understand mathematically that deep learning has more than one hidden layer, whereas regular machine learning hs just one. is that right? if so, why and how is it better to have more than one layer, that give deep learning the edge over machine learning?

Answer 1

You can think of Neural Networks (however deep) as an approximation of an ideal function. The more layers/nodes are available, the more the Neural Network successfully approximate that ideal function. There is a theorem which states it.

Let's say you have to recognize human faces in pictures: there is at least one (ideal and purely hypothetical) mathematical function that is able to do the job. Than, a Neural Network can approximate that mathematical function in a more or less precise way.

Now: if you have the collection of every possible picture which can exist in the world since ever and forever (a so-called complete set), you could choose whatever number of layers and nodes, in whatever configuration, and you will never fall in an overfit situation. Instead, the more layers/nodes are available, the better your network will perform. But in this limited real world, you can get just a tiny subset of that main set.

So you have to choose the optimal number of layers and nodes, given a configuration, to avoid to fall in the situation of overfit. The more data samples you have, the more you can add up layers and nodes to the configuration, with the result of having better performances, i.e. a Neural Network which better approximate the (ideal and purely hypothetical) mathematical function introduced above.

Answer 2

In ML research, we typically call a network with one hidden layer "shallow" and one with two or more hidden layers "deep".

If by "better" you mean the networks are more expressive and have larger capacities, then the answer is positive.

The previous answer mentioned the universal approximation theorem already. On top of that, there is a more subtle discussion of what width does not buy you when compared with depth. More explicitly, there are two ways to make neural nets more expressive:

• fix depth to be 1 (single hidden layer), enlarge width; (NTK regime)
• fix widths to be some constant, enlarge depth;

universal approx thm applies to both. Which one converges to low generalization error faster, if the two networks have the same number of total nodes? It turns out that in certain settings, one can show that the second one is preferred. Tons of references are available, I would just point to some of the brilliant discussions Disentangling feature and lazy training in deep neural networks, Width is Less Important than Depth in ReLU Neural Networks.

These provide an incentive to go deeper rather than wider. I hope the takeaway is clear that if you would like to argue that "deeper is better", then UAP alone does not explain why.

---

If by "better" you mean the networks are able to achieve better test performance, this may not be true.

I would like to push back this notion a bit. More expressive architecture unfortunately does not imply better test performance in practice. Overfitting may occur. Plus, the training dynamic may not lead you to the solution (due to gradient vanishing and many other issues): you may not be able to find a good set of parameters starting with some Gaussian initializations, even with the help of smoothing or skip connections.

Welcomes to discuss ~