1
$\begingroup$

Why do neural networks with more layers perform better than a single layer MLP with a number of neurons that leads to the same number of parameters?

I read this post:

https://www.quora.com/Why-do-neural-networks-with-more-layers-perform-better-than-a-single-layer-MLP-with-a-number-of-neurons-that-leads-to-the-same-number-of-parameters

and still I'm not sure it always true.

For example:

Assume with have model with 2 linear layers (for simplicity with no bias) and RELU as the final layer, the model look:

   RELU (w1x + w2x)

and the number of parameters to optimize is (w1 + w2)

We can see that:

  RELU (w1x + w2x) =  RELU ((x(w1 + w2)) = RELU (w3x)

so w3 = w1+w2

i.e second model with 1 linear layers has less parameters to optimize.

  1. In my example is it still better to use one layer or 2 layers ?
  2. Am I right that the second model (with w3) has less parameters?
  3. Is it easier to optimize the second model (w3) ?
$\endgroup$

3 Answers 3

1
$\begingroup$

Is bigger always better? No. Otherwise, we'd always try to build the biggest possible networks. For instance, even if larger models always have a higher accuracy, it will come at a cost. Larger models typically require more data to achieve good performance, which takes longer, requires more computing, and might be more expensive.

However

Larger/deeper neural networks can model higher-dimensional functions and, therefore, more complex problems. That's just math and not up for debate. But in practice, we don't always know how "complex" our problems are and how large of a model would be necessary. Therefore, empirical evidence typically prevails.

$\endgroup$
0
$\begingroup$

and still I'm not sure it always true.

It is not. shallow neural nets are still very much used in classical machine learning applications. Of course if you want to do something like image recognition, deeper NNs have more than proven that they work better, because there you have data that is extremely non-linear.

$\endgroup$
0
$\begingroup$

You raise a good point, the same number of parameters in a shallow network perform less than a deeper one! My theory is that it is about the optimization methodology like Backpropagation or SGD.

Also here is the AI answer

1- Feature Hierarchy: Deeper neural networks excel at constructing a hierarchy of features, starting from simple and progressing to complex.

Reference: LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444. This landmark paper describes how deep learning models, particularly convolutional neural networks (CNNs), learn a hierarchy of features, from simple to complex, through their layers.

2- Model Capacity and Expressiveness: Despite having the same number of parameters, deeper networks can leverage their depth to increase their representational power.

Reference: Goodfellow, I., Bengio, Y., Courville, A., & Bengio, Y. (2016). Deep learning (Vol. 1). MIT press Cambridge. This comprehensive textbook explains the concept of model capacity in the context of deep learning, highlighting how deeper networks can express more complex functions.

3- Optimization Dynamics: Deeper networks alter the landscape of the optimization problem in ways that can facilitate learning

Reference: Sutskever, I., Martens, J., Dahl, G., & Hinton, G. (2013). On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning (pp. 1139-1147). This paper discusses the challenges and strategies related to the optimization of deep neural networks, including the role of initialization and momentum.

4- Efficient Parameter Utilization: Deep networks can often utilize their parameters more efficiently than shallow ones for certain tasks.

Reference: Montufar, G. F., Pascanu, R., Cho, K., & Bengio, Y. (2014). On the number of linear regions of deep neural networks. In Advances in neural information processing systems (pp. 2924-2932). This work investigates the capacity of deep networks to partition input space into linear regions, illustrating the efficient use of parameters in deep architectures.

5- Implicit Regularization: Adding more layers can introduce an implicit form of regularization, as the network must propagate signals through many transformations, encouraging the learning of more generalized features.

Reference: Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958. Although this paper primarily introduces dropout as a regularization technique, it discusses the concept of regularization in deep learning, including implicit forms such as layer depth.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.