I am thinking a simple question which just came to my mind.

There are many people using data augmentation on their image data to train deep CNN.

When I learn from Andrew Ng's DL courses, he mentioned that to train a better model, you generally need to have your train/test data come from the same distribution. Like, if you train you CNN with a lot of images of cars and human, it's (usually) not a good idea to use it to classify a cat or a dog.

After the augmentation, my training data is kind of boosted, then can I still say my train/test data set comes from the same distribution? Am I violating what he said? Why? Or, why not?


3 Answers 3


I would argue that augmentation does not alter your train/test sets distribution.

I say this because I consider data augmentation to be part of your training pipeline and not part of your train set.

The pipeline below is one that I use often when someone asks about data augmentation. At the moment the data is split into train/test(/validation) sets, each set should have the same, or at least very similar distributions. This is the moment you need to worry about by picking a good splitting strategy.

Then, whatever you do with the train set, including data augmentation is part of your training pipeline. You will do data augmentation, like mentioned by others, do improve you model robustness. The validation and test set, do not need to be passed through your augmentation process.

          +-> training set ---> data augmentation --+
          |                                         |
          |                                         +-> model training --+
          |                                         |                    |
all data -+-> validation set -----------------------+                    |
          |                                                              +-> model testing
          |                                                              |
          |                                                              |
          +-> test set --------------------------------------------------+
  • $\begingroup$ But effectively the training set is increased with data augmentation. So how do you know that the training distribution is not altered? $\endgroup$
    – ado sar
    Mar 11 at 12:19
  • $\begingroup$ @adosar you control the data augmentation, if you want to keep the same distribution, then you add training samples reflecting that distribution. Alternatively, you can also balance your classes. $\endgroup$ Mar 13 at 8:20
  • $\begingroup$ Alright, that makes sense! For example, if our distribution does not contain upside down images (e.g an image of a face rotated 180 degrees), then we should not add such extreme transformations right? But if such images exist in our dataset, then we can (and is advisable) to add such upside down images with augmentation. $\endgroup$
    – ado sar
    Mar 14 at 12:38

I don't know how to put this but your understanding of distribution is wrong and confusing you. Data Augmentation is a technique which helps the model generalize well. It is also useful when you have a limited amount of data and you still want to train a deep neural net. Transfer learning is the best example of that. And yes train/test should come from the same distribution. For example, and it makes sense if you put this thing in the real world too. If you are trained to speak English, would it make sense to evaluate you on how good you speak German?

Does augmentation change the distribution? I will ask you a question on this and you should reason that for yourself. Here is my question: If you rotate the image of a dog by 30 degrees, does it change the fact that it's a dog?

  • $\begingroup$ Yeah it's still a dog. I know it's for model generalization. But during the augmentation people usually add Gaussian, Salt and Pepper noise to the image (and those noise seems not possible for a object detector?)... Which is almost impossible to see during the test time for the object detector or classifier. $\endgroup$
    – Kulbear
    Nov 2, 2017 at 15:59
  • $\begingroup$ The whole purpose of adding noise during training is to make the model more robust. Adding Pepper isn't changing the dog. $\endgroup$
    – enterML
    Nov 2, 2017 at 16:01
  • $\begingroup$ You should only add noise (blur or whatever) if the user images might come with noise as well. If you know for sure that user images are crystal-clear, then no need to train with noisy images. $\endgroup$ Jan 20, 2018 at 22:26
  • $\begingroup$ @enterML What is the point of rotating an image if all images in the data generating distribution have a fixed orientation? Doesn't data augmentation in this scenario effectively alters the distribution of the training set? Also a 180 degrees rotation of a dog image doesn't change the fact that is a dog, but why we don't apply such strong rotations? $\endgroup$
    – ado sar
    Feb 23 at 17:33

After the augmentation, my training data is kind of boosted, then, can I still say my train/test data set come from a same distribution?

In my opinion, yes you can say they still come from the same distribution. As far as I understood, what data augmentation does is not an alteration of the distribution of your data, but rather a better exploration of it.

You could think of your training set, theoretically, as a sample from a hypothetical distribution that you are trying to understand and explore. Data augmentation provides your model with more information about how that hypotetical distribution is shaped.

That is why, IMHO, data augmentation is such a powerful tool in fighiting overfitting, for the improvement of you model's performance. Its robustness comes out greatly increased.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.