Is data augmentation changing the train/test sets distribution?

Question

I am thinking a simple question which just came to my mind.

There are many people using data augmentation on their image data to train deep CNN.

When I learn from Andrew Ng's DL courses, he mentioned that to train a better model, you generally need to have your train/test data come from the same distribution. Like, if you train you CNN with a lot of images of cars and human, it's (usually) not a good idea to use it to classify a cat or a dog.

After the augmentation, my training data is kind of boosted, then can I still say my train/test data set comes from the same distribution? Am I violating what he said? Why? Or, why not?

Answer 1

I would argue that augmentation does not alter your train/test sets distribution.

I say this because I consider data augmentation to be part of your training pipeline and not part of your train set.

The pipeline below is one that I use often when someone asks about data augmentation. At the moment the data is split into train/test(/validation) sets, each set should have the same, or at least very similar distributions. This is the moment you need to worry about by picking a good splitting strategy.

Then, whatever you do with the train set, including data augmentation is part of your training pipeline. You will do data augmentation, like mentioned by others, do improve you model robustness. The validation and test set, do not need to be passed through your augmentation process.

          +-> training set ---> data augmentation --+
          |                                         |
          |                                         +-> model training --+
          |                                         |                    |
all data -+-> validation set -----------------------+                    |
          |                                                              +-> model testing
          |                                                              |
          |                                                              |
          +-> test set --------------------------------------------------+

Answer 2

I don't know how to put this but your understanding of distribution is wrong and confusing you. Data Augmentation is a technique which helps the model generalize well. It is also useful when you have a limited amount of data and you still want to train a deep neural net. Transfer learning is the best example of that. And yes train/test should come from the same distribution. For example, and it makes sense if you put this thing in the real world too. If you are trained to speak English, would it make sense to evaluate you on how good you speak German?

Does augmentation change the distribution? I will ask you a question on this and you should reason that for yourself. Here is my question: If you rotate the image of a dog by 30 degrees, does it change the fact that it's a dog?

Answer 3

After the augmentation, my training data is kind of boosted, then, can I still say my train/test data set come from a same distribution?

In my opinion, yes you can say they still come from the same distribution. As far as I understood, what data augmentation does is not an alteration of the distribution of your data, but rather a better exploration of it.

You could think of your training set, theoretically, as a sample from a hypothetical distribution that you are trying to understand and explore. Data augmentation provides your model with more information about how that hypotetical distribution is shaped.

That is why, IMHO, data augmentation is such a powerful tool in fighiting overfitting, for the improvement of you model's performance. Its robustness comes out greatly increased.