When is a Neural networked considered to be overfitted?

Question

When is a neural network considers to be overfitted to a specific task.

I read somewhere that if the number of input parameter equate the number of hyper parameter, that would be an indication of overfitting..

I use CV for training. first i split the data into train/test => 90/10. and within the training set ,i do another 90/10 split, which are those data I use for the training.

I then use the model for processing the complete dataset.

CV would state how much i am overfitting (I am getting minor error, deviating with 2%).. But since i am using most of the data for training? am I then creating an overfitted neural network for this dataset.

I am currently having 230043 training samples, 107145 total params, and validates on 25561 samples. am I overfitting?

Answer 1

A model is said to have overfitted when it performs awesomely on the training set (error's low), but when tested on test set, it fails badly. This might be due to bad sampling distribution over our training and tes divisions, so that model might not have seen this kind of data or due to using many irrelevant features in the samples than we were supposed to etc.

You cannot really say that the cat is dead until you open the box. So until you test, you would never know.

I read somewhere that if the number of input parameter equate the number of hyper parameter, that would be an indication of overfitting

This statement can be explained in general, not only for neural nets. Assume you know the relationship is linear for your data, but you got some error and you are not happy with this, then say you add an extra term say $ax^2$ to this equation and the model now has an extra parameter to tune and this makes our line into curve. For a neural net we basically start by random weights from a distribution and then we keep reducing them until we feel like our training error is cool.

Assuming that you are using a multi-layer perceptron (fully connected), pruning is one of the techniques to remove weights from the model which might not be so contributing still making sure your model still behaves well with lesser number of parameters. SVD on this weight matrix can be one of the useful techniques to do this. But how many weights to keep depends on how many singular values you keep for reconstructing the matrix. I heard that early stopping can reduce overfitting, but I had never really done this, so can't really give my opinions about this.

Hope this helps.

Answer 2

The risk of overfitting increases with the number of parameters, but there are techniques to limit overall model complexity so that the number of parameters can still be higher than the number of samples. In general they are called regularization. Implementation depends on the type of model - for neural nets, commonly used methods are weight decay, L1/L2 regularization and dropout.

In fact most non-linear models are slightly overfitted in practice. As long as you cross-validate properly and choose the model only based on out-of-sample performance, this is not a major concern. If your model overfits badly (train error is significantly lower than test error), increase regularization to get a potential performance gain.