I want to train my neural net to overfit the training data. Should I just keep fitting my model with the same training data by using K-fold validation and setting epochs to infinity? Then, after it reaches nearly 100 percent prediction, I can just stop the training. Is it the right way to do it?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
5
If you want to overfit, then yes you just need to keep fitting the training data through your network until you reach as close to zero training loss as possible (note that zero loss is stronger than 100% prediction, and will result in a greater amount of overfitting). There is no need to use cross validation. If your network is sufficiently complex, then it won't take very long to reach very high accuracy on the training data.
-
$\begingroup$ I actually try to split my training data into two part one for fitting the model another is for testing. Then I keep fitting model with same data and using the same testing data over and over again but my loss function is not decrease that fast or not it is not decrease at all. It is around 1.8 and 1.7. I have like 12 layer of convolution neural net. with kernel of size 5 by 5 on each of the layers. Do you have any suggestion for me to look for? $\endgroup$– MakaraPrCommented Dec 8, 2016 at 5:42
-
$\begingroup$ @MakaraPr - it is not clear which evaluation of the loss function is not decreasing? Are you looking at the loss over the test data? $\endgroup$ Commented Dec 8, 2016 at 11:57
-
$\begingroup$ Yes, the loss function and also the f1_score is not decreasing at all it keep at around 0.12 or 0.13 that is it. $\endgroup$– MakaraPrCommented Dec 8, 2016 at 11:59
-
3$\begingroup$ @MakaraPr: If you were over-fitting I would expect the loss function on the test data to increase. Over-fitting means your training loss decreases, whilst the test loss does not improve and typically will increase. $\endgroup$ Commented Dec 8, 2016 at 20:04
-
$\begingroup$ You need a validate set because sometimes a model is not complex enough to overfit on the data. In that case both training and validate loss will stop decreasing but validate loss doesn't increase. $\endgroup$ Commented Jan 10, 2018 at 10:41
$\begingroup$
$\endgroup$
To deliberately over fit neural network , give stopping threshold and error metric to zero and let the neural network run for huge number of iteration until zero error is reached .
And do not perform any regularization , input or hidden layer drop out , cross validation
$\begingroup$
$\endgroup$
5
It is very important to understand the difference between overfitting / underfitting and bias error / variance error.
Three things to have high probability to overfitting:
- Complex model with very large number of parameters , there is a relation between the number of samples used for training and the number of parameters. Read about optimism
- Few number of training samples
- Training for large number of iterations, until you have training error very close to zero
It is very important to know that these things increase the probability to have overfitting, but another important factor is the problem itself. If you have two classes that are separable using your feature space, even with all these conditions you may not overfit, even if the training error is zero.
answered Dec 8, 2016 at 5:06
-
$\begingroup$ my input data set is three dimensional array of size 23 by 23 by 35 with my constitutional neural net with 12 layer with kernel size of 5 by 5 for the first on and 3 by 3 for the rest. There are about 3500 dataset for this training. I put it down to train for a day but the f1_score is still around 0.1284 not increase what so ever. Do you have any other suggestion what to look for? $\endgroup$– MakaraPrCommented Dec 8, 2016 at 7:45
-
1$\begingroup$ Go for broader networks instead of deeper. More kernels per layer are very good in overfitting since they tend to learn specific examples and generalize less well. But again, why do you even want overfitting? Could you explain what your goal is of getting an overfit? $\endgroup$ Commented Dec 8, 2016 at 11:03
-
1$\begingroup$ There is a trick you can use. I avoided talking about it because it uses usually for testing for overfitting instead of overfitting. Try to introduce some noise, what you can do is to select some samples from each class and change its label. These noisy labels will enforce the network to overfitting. The question asked is why you want to have overfitting? Usually it is an issue we try to avoid not achieve $\endgroup$ Commented Dec 8, 2016 at 11:45
-
$\begingroup$ @Laurens I want to overfit my training data because I want to make sure that my neural net has sufficiently enough parameter for training my data input with its prediction. $\endgroup$– MakaraPrCommented Dec 8, 2016 at 11:53
-
1$\begingroup$ you can add more kernel for each layer of the net $\endgroup$– MakaraPrCommented Dec 8, 2016 at 11:54