Approximating density of test set in train

Question

I am looking for a method to approximate how similar a test set (i.e., test set features) to a train set. For example, something like, for each row in test: is there a similar enough data point in train? I've been thinking about using a mixture model approach, but I haven't been able to find a good reference on this. Can anyone suggest a good approach, or provide good references for how to use mixture models for this application?

Answer 1

The approach that comes to mind, is to calculate the kullback-leibler divergence between the kernel density estimations of your train dataset and of your test dataset.

The kernel density estimation of each of your datasets will give you an approximation to the pdf's of your datasets. The kullback-leibler divergence will give you a number that will represent the divergence in bits from one distribution to another (if you use base 2 for your logarithm). Below are some references I think you would fine useful.

https://en.wikipedia.org/wiki/Kernel_density_estimation https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/

If you would like me to show the math behind this method. Feel free to ask.

EDIT: Added math as asked by author of question

Let $\hat x_1, \hat x_2,\hat x_3...,\hat x_n$ be your training dataset while $x_1,x_2,x_3...,x_n$ is your testing dataset, where both $\hat x_i$ and $x_i$ belong to $\mathbb{R}^d$. $$\hat f(x;H)=\frac{1}{n} \sum_{i=1}^{n}K(x-\hat x_i;H)$$ $$f(x;H)=\frac{1}{n} \sum_{i=1}^{n}K(x-x_i;H)$$ $\hat f$and $f$ represent the kernel density estimation for training set and testing set respectively. The parameter $H$ represents the bandwidth parameter and is a symmetric positive definite $d \times d$ matrix. $K(u;H)$ can be rewritten as $$|H|^{-\frac{1}{2}} K(H^{\frac{1}{2}} u)$$ where K can be any kernel function. I would recommend for simplicity purposes the standard multivariate normal kernel. Okay so now that we have the kernel density estimations of both our training and our testing dataset, we can use the kullback-leibler divergence in order to estimate the difference between the two.
Optimally we would like to calculate the kullback-leibler divergence with respect to every point in our space. Mathematically speaking.

$$\int_X f(x;H) \ log_2(\frac{f(x;H)}{\hat f(x;H)})dx$$

But this is computationally unpractical to compute. We can approximate this integral by sampling a set of points from the $f(x;H)$ and then computing the discrete sum. $$\sum_{x \in X} f(x;H) \ log_2(\frac{f(x;H)}{\hat f(x;H)})$$

Quick Note: To sample from a kernel density estimator, uniformly randomly select a point from the respective dataset. Then sample from the kernel of choice centered around the point chosen.