8
$\begingroup$

A basic assumption in machine learning is that training and test data are drawn from the same population, and thus follow the same distribution. But, in practice, this is highly unlikely. Covariate shift addresses this issue. Can someone clear the following doubts regarding this?

How does one check whether two distribution are statistically different? Can kernel density estimate (KDE) be used to estimate the probability distribution to tell the difference? Let's say I have 100 images of a specific category. The number of test images is 50, and I'm changing the number of training images from 5 to 50 in steps of 5. Can I say the probability distributions are different when using 5 training images and 50 testing images after estimating them by KDE?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Please don't cross post (stats.stackexchange.com/questions/173968/…) $\endgroup$
    – Dawny33
    Commented Sep 25, 2015 at 3:16
  • $\begingroup$ @Dawny33: It seems this question is more relevant to this site than cross validated. That's why I posted here. $\endgroup$
    – Daniel Wonglee
    Commented Sep 25, 2015 at 3:52
  • $\begingroup$ This is a tough one for two reasons. If turning the images into a distribution using a KDE were viable, I'd tell you to apply a two sample Kolmogorov–Smirnov test. But, the two dimensional nature of the image will render this difficult. Also there is a tiling effect in images that won't be recovered well with K-S. I thus suggest image processing: Haussdorff distance. Also check out this post. $\endgroup$
    – AN6U5
    Commented Dec 17, 2015 at 17:54

2 Answers 2

Reset to default
1
$\begingroup$

A good way to measure the difference between two probabilistic distributions is Kullbak-Liebler. You have to take into account that the distribution has integrate to one. Also you have to take into account that it's not a distance because it's not symmetric. KL(A,B) not equal to KL(B,A)

Improve this answer
$\endgroup$
0
$\begingroup$

If you are working with large dataset. Training and test set disribution may not be too different. In theory "law of large numbers" ensures that the distribution remains same. For smaller set of data probably this is a good point to take care of distribution. As said by Hoap Humanoid "Kullbak-Liebler" can be used to find the difference of distributions of two sets.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I'd add that stratification could be a viable tool to deal with smaller sets once acknowledge that the distance is above your threshold. $\endgroup$
    – pincopallino
    Commented Jan 25, 2016 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.