# Difference between training and test data distributions

A basic assumption in machine learning is that training and test data are drawn from the same population, and thus follow the same distribution. But, in practice, this is highly unlikely. Covariate shift addresses this issue. Can someone clear the following doubts regarding this?

How does one check whether two distribution are statistically different? Can kernel density estimate (KDE) be used to estimate the probability distribution to tell the difference? Let's say I have 100 images of a specific category. The number of test images is 50, and I'm changing the number of training images from 5 to 50 in steps of 5. Can I say the probability distributions are different when using 5 training images and 50 testing images after estimating them by KDE?

• Please don't cross post (stats.stackexchange.com/questions/173968/…) Sep 25, 2015 at 3:16
• @Dawny33: It seems this question is more relevant to this site than cross validated. That's why I posted here. Sep 25, 2015 at 3:52
• This is a tough one for two reasons. If turning the images into a distribution using a KDE were viable, I'd tell you to apply a two sample Kolmogorov–Smirnov test. But, the two dimensional nature of the image will render this difficult. Also there is a tiling effect in images that won't be recovered well with K-S. I thus suggest image processing: Haussdorff distance. Also check out this post. Dec 17, 2015 at 17:54

## 2 Answers

A good way to measure the difference between two probabilistic distributions is Kullbak-Liebler. You have to take into account that the distribution has integrate to one. Also you have to take into account that it's not a distance because it's not symmetric. KL(A,B) not equal to KL(B,A)

If you are working with large dataset. Training and test set disribution may not be too different. In theory "law of large numbers" ensures that the distribution remains same. For smaller set of data probably this is a good point to take care of distribution. As said by Hoap Humanoid "Kullbak-Liebler" can be used to find the difference of distributions of two sets.

• I'd add that stratification could be a viable tool to deal with smaller sets once acknowledge that the distance is above your threshold. Jan 25, 2016 at 13:59