According to my professor one of the first steps in modelling a NN is to use a powerful enough model.

The first step is to create a model that is powerful enough to achieve very high accuracies (very low loss) on the training data, at least when no regularisation is used.

What are some of the things (obviously apart of regulazing and adjusting learning rate), that I can do to make my model "powerful" enough, in other words to let it overfit on the training data?

Am I looking in the right direction with the following things?

  1. Add extra layers
  2. Make layers thicker (more neurons)
  • $\begingroup$ If you are aiming to overfit, (I assume with the intention to then reduce this later) you should look to create a more 'complex' model. i.e. more flexible i.e. more parameters. This means no regularization / dropout. No early stopping. More layers and more nodes. $\endgroup$ – Dan Feb 25 at 13:44
  • $\begingroup$ Yes the intention is to reduce the overfitting later using regularization, early stopping, ... But the first step should be to have a complex/powerful enough model that is able to overfit on the training data. The professor indicated that when plotting the Accuracy (Y-axis) vs the epochs (X-axis), the accuracy should quickly go to 1. Apart from how this is done, do you have any explanation on why? And could I play around with the learning rate for this, or should I also stay away from this (like the regularization)? $\endgroup$ – Tibo Geysen Feb 25 at 13:47
  • $\begingroup$ Overfitting means modelling noise. The more parameters your model has, the more flexibly it can learn each point of noise. Think about a linear model, it can't go through each noisy point so it will struggle to overfit noisy data. But if you have a polynomial model, for each new polynomial term you add (i.e. a new model parameter to learn), you give it more flexibility to bend to fit each point. In a NN, the weights are the parameters, adding more nodes means more weights mean it can model more noise. $\endgroup$ – Dan Feb 25 at 13:51
  • $\begingroup$ Can I assume that to overfit, the amount of parameters it thus important? The way you get this is less important (thicker or deeper network)? $\endgroup$ – Tibo Geysen Feb 25 at 17:00

Here are some general hyperparameter adjustments to increase model capacity:

  1. Architecture-related:

    • add additional layers
    • increase the number of hidden units per layer
    • remove dropout or decrease dropout rates
  2. Optimizer related:

    • find the optimal learning rate
    • train for more epochs

Section 11.4.1 in the Deep Learning Book provides a good overview. The paper "Practical recommendations for gradient-based training of deep architectures" is another good source to understand the effect of different hyperparameters (though it is a bit older).

Moreover, there are specific hyperparameters to increase model capacity for different network types, e.g.

  • kernel size for CNNs,
  • sequence length for LSTMs and
  • embedding dimension for embedding layers.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.