What does one mean when they are talking about the simplicity of the networks? Does it mean that the shallower the networks the simpler they are, or does it mean that lesser the number of trainable parameters, simpler the models?
2
-
1$\begingroup$ Could you clarify the link between your title and your question? $\endgroup$– LeevoCommented Dec 1, 2019 at 9:27
-
$\begingroup$ I want to understand whether the depth of pre-trained neural networks makes them complex or the number of parameters. Sorry for the misleading title. I will change it. $\endgroup$– Piyush DongreCommented Dec 1, 2019 at 23:00
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It can mean both, depending on the context. In order to improve prediction, a Network might need either more parameters or more depth, but these features improve models in different ways. For example, it's true that a Network with too few parameters won't be able to learn much; there is a minimal amount of parameters required simply to transform and represent signals in a sufficiently sofisticated way. On the other side, depth is fundamental in order to make your model generate more complex abstractions that will make it superior to other ML algorithms.
In other words: the number of parameters overall tells you how many things your network can learn. Its depth tells you how complex and sofisticated these things can be.
Deep Neural Network complexity is a multifaceted thing, it can't be represented on a single dimension.
