I am looking for a method to approximate how similar a test set (i.e., test set features) to a train set. For example, something like, for each row in test: is there a similar enough data point in train? I've been thinking about using a mixture model approach, but I haven't been able to find a good reference on this. Can anyone suggest a good approach, or provide good references for how to use mixture models for this application?


1 Answer 1


The approach that comes to mind, is to calculate the kullback-leibler divergence between the kernel density estimations of your train dataset and of your test dataset.

The kernel density estimation of each of your datasets will give you an approximation to the pdf's of your datasets. The kullback-leibler divergence will give you a number that will represent the divergence in bits from one distribution to another (if you use base 2 for your logarithm). Below are some references I think you would fine useful.

https://en.wikipedia.org/wiki/Kernel_density_estimation https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/

If you would like me to show the math behind this method. Feel free to ask.

EDIT: Added math as asked by author of question

Let $\hat x_1, \hat x_2,\hat x_3...,\hat x_n$ be your training dataset while $x_1,x_2,x_3...,x_n$ is your testing dataset, where both $\hat x_i$ and $x_i$ belong to $\mathbb{R}^d$. $$\hat f(x;H)=\frac{1}{n} \sum_{i=1}^{n}K(x-\hat x_i;H)$$ $$f(x;H)=\frac{1}{n} \sum_{i=1}^{n}K(x-x_i;H)$$ $\hat f$and $f$ represent the kernel density estimation for training set and testing set respectively. The parameter $H$ represents the bandwidth parameter and is a symmetric positive definite $d \times d$ matrix. $K(u;H)$ can be rewritten as $$|H|^{-\frac{1}{2}} K(H^{\frac{1}{2}} u)$$ where K can be any kernel function. I would recommend for simplicity purposes the standard multivariate normal kernel. Okay so now that we have the kernel density estimations of both our training and our testing dataset, we can use the kullback-leibler divergence in order to estimate the difference between the two.
Optimally we would like to calculate the kullback-leibler divergence with respect to every point in our space. Mathematically speaking.

$$\int_X f(x;H) \ log_2(\frac{f(x;H)}{\hat f(x;H)})dx$$

But this is computationally unpractical to compute. We can approximate this integral by sampling a set of points from the $f(x;H)$ and then computing the discrete sum. $$\sum_{x \in X} f(x;H) \ log_2(\frac{f(x;H)}{\hat f(x;H)})$$

Quick Note: To sample from a kernel density estimator, uniformly randomly select a point from the respective dataset. Then sample from the kernel of choice centered around the point chosen.

  • $\begingroup$ Hi , sure the math behind will be so good if you are kind enough . The only thing I dont understand is how cand I make the kernel density estimare for all the feature spaces between the 2 sets? $\endgroup$
    – gm1
    Commented Jan 8, 2016 at 21:12
  • $\begingroup$ @gm1 Did the edits to my answer, answer your question? $\endgroup$ Commented Jan 9, 2016 at 9:05
  • $\begingroup$ Yes great :) I selected an upvoted! $\endgroup$
    – gm1
    Commented Jan 9, 2016 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.