1
$\begingroup$

I trained different classification models using Keras with different numbers of hidden layers and the same number of neurons in each layer. What I found was the accuracy of the models decreased as the number of hidden layers increased, however, the decrease was more significant in larger numbers of hidden layers. The following graph shows the accuracy of different models where the number of hidden layers changed while the rest of the parameters stay the same (each model has 64 neurons in each hidden layer): enter image description here

My question is why is the drop in accuracy between 8 hidden layers and 16 hidden layers much greater than the drop between 1 hidden layer and 8 hidden layers, even though the difference in the number of hidden layers is the same (8).

$\endgroup$
3
  • 1
    $\begingroup$ short answer: the increase of complexity is similar to exponential as hidden layers increase $\endgroup$
    – Nikos M.
    Apr 10 at 16:16
  • $\begingroup$ @NikosM. thanks for answering, could you cite a paper or include the name of this behaviour so I could search more about it? $\endgroup$
    – Shonix3373
    Apr 11 at 6:30
  • $\begingroup$ the area involves complexity of NNs in relation to their performance or predictive power. Some references I could find are: papers.nips.cc/paper/2014/file/…, math.bu.edu/people/mkon/pppp11.pdf $\endgroup$
    – Nikos M.
    Apr 11 at 9:35
1
$\begingroup$

Neural Networks try to approximate the function which maps the given image to its label. Changing the number of hidden layers essentially means changing the number of parameters, which would be used to approximate that input-label mapping. These parameters include weights and biases ( in the case of Dense layers ). Depending upon the number of parameters, the model might,

  • Underfit, meaning it has lesser parameters than that required to solve the problem. This would result in a decrease in the model's accuracy over the test dataset.

  • Overfit, meaning the model has excessive parameters and now it tends to memorize the given training dataset. In this case, the model's accuracy over the training dataset would be much higher than that over the testing data.

Apart from these cases, if the model's loss, as well as accuracy ( or any metric to measure the performance of the model ), seems to improve both on the testing and training data, up to a certain number of epochs, we say that the model has generalized itself.

It depends on the problem you're trying to solve. CNNs like InceptionV3, MobileNets, ResNets have hundreds of hidden layers as they are trained on huge image datasets like CelebA or ImageNet. So, they require millions of parameters to generalize themselves and obtain a better accuracy over the testing data.

You may find a clearer definition of overfitting here.

My question is why is the drop in accuracy between 8 hidden layers and 16 hidden layers much greater than the drop between 1 hidden layer and 8 hidden layers, even though the difference in the number of hidden layers is the same (8).

Probably the no. of parameters in those hidden layers were just up to the mark for the model to generalize itself. As the no. of hidden layers was increased to 16, the model had excessive parameters and it overfitted the training data.

How do we determine the no. of hidden layers required by a model?

One way to determine is to perform hyperparameter optimization and to search for the best possible combinations of hyperparameters ( learning rate, dropout rate ).

$\endgroup$
2
  • $\begingroup$ Thank you for answering, does this mean that up until 8 hidden layers the model didn't overfit? Because the accuracy did decrease but much less than it did with 16 hidden layers. $\endgroup$
    – Shonix3373
    Apr 11 at 6:28
  • $\begingroup$ Yes @Shonix3373, you can have such an interpretation. $\endgroup$ Apr 11 at 7:25
0
$\begingroup$

As you add more parameters, your model is probably overfitting to the training data, that is, it is memorizing the training data and therefore is worse at generalizing when used on the test/validation data.

$\endgroup$
2
  • $\begingroup$ Thanks for answering, the model's accuracy decreased between 1 and 8 hidden layers as well as 8 and 16. does this mean the model was overfitting from the start? if so, why did the accuracy drop more drastically between 8 and 16 hidden layers? $\endgroup$
    – Shonix3373
    Apr 11 at 6:32
  • 1
    $\begingroup$ In order to better assess how overfitting was affecting each configuration, we should have a look at the training and validation loss curves, we you can clearly see the training dynamics. $\endgroup$
    – noe
    Apr 11 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.