I was self-teaching myself. I totally understand why depth of a neural network affects the learning and how it differs than its width. But I am looking for some theoretical justification about it. Papers I could come up with, e.g., Benefits of depth in neural networks or The Power of Depth for Feedforward Neural Networks are unfortunately is too deep and long. I am not also super good at Mathematics, however, I believe there must be a plain, short and compact math behind it. Can someone point me to some tutorials/articles/papers/reports, where I can easily understand it?
I think the short mathematical intuition would be to show that Deeper offers more flexibility.
Imagine we want to fit an impossibly complicated function, like this:
... but in an n-dimensional space - we obviously cannot visualise such a function. But we can agree is is complicated
Basic math analogy
The goal of the neural network would be to map the raw input data (e.g. images to a convolutional network) to some output, by approximating the complicated function.
So if we have some input, and apply a non-linear function $f$ to it, we transform it into something else:
$$output_1 = f(input)$$
Perhaps that gave us a curvy function, but it doesn't get close to matching the complicated function, so we endow the model with another chance, by applying a second non-linear function:
$$output_2 = g(output_1)$$
Giving a second function is almost like offering another degree of freedom (or flexibility) to the model.
We continue like this until the chain of non-linear functions is able to map out the output space sufficiently well. Perhaps we end up with this:
$$output_6 = k(j(i(h(g(f(input)))))$$
In this framework, imagine every single non-linear function as one of the layers in a deep network. The deeper the network gets, the more functions we are applying and the more we mould and transform the input to something else; perhaps in different ranges, by different magnitudes and so on.
This should (in a very hand-wavey manner) convince you that more functions applied gives more possibilities in the final output space. So more layers gives us more power to express more and more complicated functions.
A practical note: the more layers we add, the more powerful the model and the larger the tendency to either:
- be very difficult to train
- eventual overfit the training data completely
- the longer the model takes to train
- the greater the level of regularisation that is likely required to obtain a reasonable validation metric on unseen data
I recommend the use of the Tensorflow playground to get an intuitive understanding. In particular, try to find a fit for the spiral dataset.
The way I understand it, each layer of a neural network can be composed of all the shapes provided by all the previous layers. The first layer (assuming sigmoid or tanh activation) can only make sigmoid shapes. Each node in the second layer can now make complicated shapes out of linear combinations of all the sigmoid shapes in the first layer. Each node in the third layer can make even more complicated shapes out of the complicated shapes of the second layer. And so on, and so forth.