I tried to fit my model on a small batch of 128 samples for binary classification. The model should be powerful enough as it has hundreds of thousands of parameters. It should be able to overfit to 100% accuracy. However, it only fts to 96% for the best. It is about the same as when I train it on 30,000 samples. So, I tried the following but all failed:

use a smaller batch of 16 samples, it still cannot overfit

use different optimizers, including Adam, SGD, Adagrad, even reset the optimizer every 1,000 epochs, not working

every epoch, only train on the samples that are misclassified, not working.

The problem should be with this network since another more basic neural network can 100% fit. This one can only 99.2% fit. The top layer is indeed sigmoid.

Anyone got any idea what could be the problem?

  • $\begingroup$ How many features does the data have? $\endgroup$
    – PSub
    Mar 19, 2021 at 3:18
  • 1
    $\begingroup$ I'd check if the signal contained in the data actually allows perfect classification. For example, 2 instances w/ identical feature values could belong to differenct classes. If it's tabular data just apply an unpruned decision tree and examine potentially unpure leaves. $\endgroup$
    – Jonathan
    Mar 19, 2021 at 8:23
  • $\begingroup$ Should easily overfit. You might have a few duplicate instances with opposite True label. $\endgroup$
    – 10xAI
    Mar 19, 2021 at 11:17
  • $\begingroup$ The problem should be with the network since another more basic neural network can 100% fit. This one can only 99.2% fit. $\endgroup$ Mar 19, 2021 at 12:10
  • $\begingroup$ @PSub Actually, it has tens of thousands of features $\endgroup$ Mar 29, 2021 at 20:40

3 Answers 3


I would guess there are a few samples that aren't necessarily from the same distribution as the rest. I would try identifying outliers and removing them.


Although it is true that you have more parameters than samples, DNN trains all those parameters at the same time, which makes it harder to overfit. Try reducing the learning rate and use SGD with momentum = 0. In addition, don't forget to remove any type of regularisation.

I am assuming you want to keep using 128 samples and the network you have designed, but you can always reduce the number of parameters or use a standard network to test (ResNet, Inception, VGG). I would usually take enough samples for one batch when I want to overfit the network.

Anyhow, if the network achieves 96% I would start by reducing the learning rate.

Good luck


One way of assuring that you cannot perfectly fit in-sample is if two observations have the same values of features ($x$) but different values of the outcome ($y$). If that happens, then no function can fit perfectly, as functions map identical input values to identical output values.

If you’re in a situation where two observations have very similar (but not quite identical) feature values and different outcome values, it might be that the transition between them has to be so abrupt that you have not yet converged to such a transition.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.