3
$\begingroup$

Before data augmentation, my model clearly overfits and hits a 100% training accuracy and a 52% validation accuracy. When only adding data augmentation with Keras, as a regularization technique, it achieves a 95% training accuracy with slower convergence and a 80% validation accuracy (which is a way better result). But why does the training accuracy gets reduced by around 5%?

If somebody could provide the link to a research paper or explain the reasoning behind this, it would be greatly appreciated!

$\endgroup$
1
$\begingroup$

The obvious reasons why data augmentation might reduce the train accuracy is -

As you know, deep learning models are data hungry. If the model don't get enough data to recognize the patterns then it will try to memorize the dataset. Bigger models tend to memorize the data instead of finding patterns, because they are big enough to do so. When model memorizes the training data it will definitely perform very good on training set and poorly on validation set.

And as you said data augmentation is a regularization technique. In regularization your model weights are penalized more to make sure they don't over fit. As a result, your model cannot perform well on training set (depending on how much regularzation is used), but as an advantage model will try to find generalized patterns in the dataset and this will also help at the time of validation.

I could find one research paper which has exhaustive experiments about data augmentation and regularization.

$\endgroup$
2
  • $\begingroup$ Thank you very much for the answer! That might be some of the best reasons for this behavior. The paper was an interesting read, but I could not find much information about how it affects accuracy on the training set. However, why would this (adding data augmentation) also imply slower convergence then? Is that due to having more data while training? $\endgroup$
    – Kralley
    May 29 at 8:41
  • $\begingroup$ Your welcome @Kralley, Yes slower convergence will happen because model is training on more data. Moreover, because of the regularizing effect, it is not easy for the model to reach minima. Just to understand the effect of regularization on training we can consider this image miro.medium.com/max/2400/0*3d9Gz6MdcwYmqZTd.png Notice, smaller batches makes it harder for model to reach convergence. Smaller batches = more regularization. Data augmentation = more regularization. And you can accept the my answer if it answers your question :) $\endgroup$ May 29 at 9:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.