7
$\begingroup$

A basic assumption in machine learning is that training and test data are drawn from the same population, and thus follow the same distribution. But, in practice, this is highly unlikely. Covariate shift addresses this issue. Can someone clear the following doubts regarding this?

How does one check whether two distribution are statistically different? Can kernel density estimate (KDE) be used to estimate the probability distribution to tell the difference? Let's say I have 100 images of a specific category. The number of test images is 50, and I'm changing the number of training images from 5 to 50 in steps of 5. Can I say the probability distributions are different when using 5 training images and 50 testing images after estimating them by KDE?

$\endgroup$
  • 1
    $\begingroup$ Please don't cross post (stats.stackexchange.com/questions/173968/…) $\endgroup$ – Dawny33 Sep 25 '15 at 3:16
  • $\begingroup$ @Dawny33: It seems this question is more relevant to this site than cross validated. That's why I posted here. $\endgroup$ – Daniel Wonglee Sep 25 '15 at 3:52
  • $\begingroup$ This is a tough one for two reasons. If turning the images into a distribution using a KDE were viable, I'd tell you to apply a two sample Kolmogorov–Smirnov test. But, the two dimensional nature of the image will render this difficult. Also there is a tiling effect in images that won't be recovered well with K-S. I thus suggest image processing: Haussdorff distance. Also check out this post. $\endgroup$ – AN6U5 Dec 17 '15 at 17:54
1
$\begingroup$

A good way to measure the difference between two probabilistic distributions is Kullbak-Liebler. You have to take into account that the distribution has integrate to one. Also you have to take into account that it's not a distance because it's not symmetric. KL(A,B) not equal to KL(B,A)

$\endgroup$
0
$\begingroup$

If you are working with large dataset. Training and test set disribution may not be too different. In theory "law of large numbers" ensures that the distribution remains same. For smaller set of data probably this is a good point to take care of distribution. As said by Hoap Humanoid "Kullbak-Liebler" can be used to find the difference of distributions of two sets.

$\endgroup$
  • 1
    $\begingroup$ I'd add that stratification could be a viable tool to deal with smaller sets once acknowledge that the distance is above your threshold. $\endgroup$ – pincopallino Jan 25 '16 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.