2
$\begingroup$

I am learning how to implement CNN, and searching on the internet I have found that a trick to design a good network is to first build it in such a way that it overfits, and then use regularization to elimnate overfitting and have a good performing network.

But how do I do this? I don't understand how do I build a network that overfits on purpose? And also in which way do I use regularization after?

Can someone helo me?

Thanks in advance.

$\endgroup$
  • 2
    $\begingroup$ I think you read something about finding a good architecture for a network, which typically goes like this: build network that is "too big" (i.e. too many layers), then remove layers until performance starts dropping. Is this what you were looking for? $\endgroup$ – Valentin Calomme Dec 5 '19 at 19:23
  • $\begingroup$ Thanks for answering. Probably yes. I don't have really a guide on how to do this, but every time this was written just in few lines, and never found a full argumentation. Removing number of layers seems a good solution, but also to make the network overfit I should not use any regularization, and then when overfitted I could add for example dropout to reduce the overfitting. What do you think about this? $\endgroup$ – J.D. Dec 5 '19 at 19:57
2
$\begingroup$

The reason of overfitting is generally because of the high(unnecessary) complexity model. So if you train a too complex(large) model, you will get a high training accuracy and low test set accuracy. Then you can start to fight with overfitting with getting more data, dropout, regularization, early stopping, global average pooling, feature scale clipping or dropping some of layers from your network.

$\endgroup$
2
$\begingroup$

It sounds like you are talking about orthogonalization, in this approach you break the focus of model fitting into 4 stages:

You build a model that:

Step 1: Fits the training data well, this is the primary focus at this point, in this step overfitting is likely to occur.

Step 2: Fits dev data well, in this step you likely to add regularization to address overfitting

Step 3: Fits test data well, this step is needed because dev data has been heavily experimented against and the model could overfit to the dev data.

Step 4: Fits real world data well

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.