I am thinking a simple question which just came to my mind.

There are many people using data augmentation on their image data to train deep CNN.

When I learn from Andrew Ng's DL courses, he mentioned that to train a better model, you generally need to have your train/test data come from the same distribution. Like, if you train you CNN with a lot of images of cars and human, it's (usually) not a good idea to use it to classify a cat or a dog.

After the augmentation, my training data is kind of boosted, then can I still say my train/test data set comes from the same distribution? Am I violating what he said? Why? Or, why not?


3 Answers 3


I would argue that augmentation does not alter your train/test sets distribution.

I say this because I consider data augmentation to be part of your training pipeline and not part of your train set.

The pipeline below is one that I use often when someone asks about data augmentation. At the moment the data is split into train/test(/validation) sets, each set should have the same, or at least very similar distributions. This is the moment you need to worry about by picking a good splitting strategy.

Then, whatever you do with the train set, including data augmentation is part of your training pipeline. You will do data augmentation, like mentioned by others, do improve you model robustness. The validation and test set, do not need to be passed through your augmentation process.

          +-> training set ---> data augmentation --+
          |                                         |
          |                                         +-> model training --+
          |                                         |                    |
all data -+-> validation set -----------------------+                    |
          |                                                              +-> model testing
          |                                                              |
          |                                                              |
          +-> test set --------------------------------------------------+

I don't know how to put this but your understanding of distribution is wrong and confusing you. Data Augmentation is a technique which helps the model generalize well. It is also useful when you have a limited amount of data and you still want to train a deep neural net. Transfer learning is the best example of that. And yes train/test should come from the same distribution. For example, and it makes sense if you put this thing in the real world too. If you are trained to speak English, would it make sense to evaluate you on how good you speak German?

Does augmentation change the distribution? I will ask you a question on this and you should reason that for yourself. Here is my question: If you rotate the image of a dog by 30 degrees, does it change the fact that it's a dog?

  • $\begingroup$ Yeah it's still a dog. I know it's for model generalization. But during the augmentation people usually add Gaussian, Salt and Pepper noise to the image (and those noise seems not possible for a object detector?)... Which is almost impossible to see during the test time for the object detector or classifier. $\endgroup$
    – Kulbear
    Nov 2, 2017 at 15:59
  • $\begingroup$ The whole purpose of adding noise during training is to make the model more robust. Adding Pepper isn't changing the dog. $\endgroup$
    – enterML
    Nov 2, 2017 at 16:01
  • $\begingroup$ You should only add noise (blur or whatever) if the user images might come with noise as well. If you know for sure that user images are crystal-clear, then no need to train with noisy images. $\endgroup$ Jan 20, 2018 at 22:26

After the augmentation, my training data is kind of boosted, then, can I still say my train/test data set come from a same distribution?

In my opinion, yes you can say they still come from the same distribution. As far as I understood, what data augmentation does is not an alteration of the distribution of your data, but rather a better exploration of it.

You could think of your training set, theoretically, as a sample from a hypothetical distribution that you are trying to understand and explore. Data augmentation provides your model with more information about how that hypotetical distribution is shaped.

That is why, IMHO, data augmentation is such a powerful tool in fighiting overfitting, for the improvement of you model's performance. Its robustness comes out greatly increased.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.