1
$\begingroup$

I've found a lot of martial on how to deal with differently distributed train/test/validation data sets, however I'm struggling to find out why they they need to have the same distribution.

Can anyone explain to me in detail why these data sets need to have the same/similar distributions? or point me toward resources that explain it?

$\endgroup$
2
$\begingroup$

You can think of the data distribution as the world that the model lives in. You want to train it to perform well in this world and to do this you have to train it on examples representing this world. Further you have to estimate its performance in this world by testing it on more examples from this world. If you train your model in one world, test and validate on another and deploy it into a third, you have no idea how well the model was trained for the real world or if it was trained for the real world at all.

Let's look at a little example. I created a little toy data set and ran regression with the same network, same optimizer and everything the same, except the initialization.

enter image description here

All 5 models give the same prediction in the domain where there is data (they even over-fit a little bit in the same way). But for x-values far away from the training data, we get widely different predictions from different models.

So what does this tell us? It tells us that our models are basically worthless for inputs that are "far" from the training data. "Far" means "from a different distribution" in this sense. But if our models are worthless for inputs that do not come from the original distribution, then any quality measures based on test and validation datasets are also worthless.

I used the word "far" a bit loosely here, basically as if I was talking about a distance. I think this is somewhat correct, but it's not the whole story. Even if the support of two distributions is the same, they can be "far" from each other if one distribution puts a lot of weight on events that the other only weighs little.

This is also a problem, because this would mean you get many samples from one distribution where the other gives only a few. The amount of training data you have for a particular input domain will influence how well your model generalizes to this domain. If now in the test or validation set you shift the bulk of the distribution, you basically over-emphasize input values that the model has hardly been trained for, while at the same time hardly testing it in domains that it has been trained for well. So again your measure of model performance does not really tell you how well your training worked.

This is why training, test and validation dataset should stem from the same distribution, but also the inputs once the model is deployed need to be from the same distribution, otherwise your quality measures and your predictions will be highly inaccurate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.