Different Test Set and Training Set Distribution

Question

I am working on a data science competition for which the distribution of my test set is different from the training set. I want to subsample observations from training set which closely resembles test set.

How can I do this?

Answer 1

Great question, this is what is known in Machine Learning paradigm as either "Covariate Shift", or "Model Drift" or "Nonstationarity" and so on.

One of the critical assumption one would make to build a machine learning model for future prediction is that unseen data (test) comes from the same distribution as training data! However, in reality this rather simple assumption breaks easily and upcoming data (its distribution) changes over time for many reasons. For those who may not be familiar with this very important problem, I encourage looking here or post!

To me, your question falls into the same category. Although I do not have the perfect solution (an implementation to offer), but I think you may look:

• This blog post gives you a simple way to handle the subsampling of training data with code provided in Python!
• Check this research paper. They propose to solve the problem by reweighting the training data so that the distribution of training is closer to the distribution of test using Kullback-Leibler Importance Estimation Procedure base on "Kullback-Leibler divergence" theorem. I do not know if they provide an implementation or it can be implemented easily, but I think it might worth digging as it sounds a professional way to deal the distribution mismatch.

QUICK update (a good solution): I found a Python implementation of KLIEP algorithm of that research paper (last point) to find those weights. It rather seems easy to use! Basically it resamples the training by putting weights (via the KLIEP algorithm) so that the assumption of having a similar distribution of train and test holds true as much as possible.

Answer 2

I want to subsample observations from training set which closely resembles test set.

I am not sure you'd want to do that. The whole purpose is rather to train your algorithm so that it generalises well to unseen data.

Usually, one should adapt its test data to its train data (e.g. standardising test data according to train data) and not the other way around. In practice, you don't know your test data.

Answer 3

Train set subsampling might not be the best solution!

The differences between test/execution set and training set distribution/features are very common in supervised learning tasks (this is one of the reasons that competitions such as Kaggle are challenging). That is why we say the past performance may be (only) used as a guide for estimating the future performance but it does not indicate/guarantee it. Therefore, generalizable models have always been preferred over fine-tuned models that may perform very well on the train (sub)set but do poorly on unseen data.

While such difference is normal, the too large gap between the past and future sample may be referred as examples of concept drift which is an active research field by itself. Given your question, I cannot judge that your case is a normal ML case or the concept drift is really happening.

These are my suggestions:

1. Train a number of models with high generalization capability. Using bootstrap sampling from your train dataset, you can easily calculate bias and variance components of errors. Recall that you are looking for a low-variance model (where the changes in data would have a marginal effect on its performance) rather than low-bias but high-variance models (that might overfit to your training (sub)set). Now, you can select the best algorithms and evaluate them against the test set. Note that in the training time we supposed to not look at the test set!

2. Instead of several random downsampling, look for standardization/normalization and feature selection/engineering. These techniques might be practical in learning more general models. For example, sometimes the range of feature domain may change over time while the shape of distribution (whatever it is) remains almost the same (eg same distribution that is shifted towards left or right). In such case, a simple standardization (ie mapping the train and test samples to a predefined space such as [0,1] using different mapping functions) can reduce the symptoms.

3. Systematic downsampling can only be an appropriate solution if you do it based on some knowledge about the problem (not just for the purpose of getting a better accuracy on the test dataset). For example, you might know that some of the records in the train data are sampled a long time ago, from far field, or affected by particular factors which none of them will happen in future (in test data collection). In such case, you may remove those samples that can be irrelevant as you are confident that you will not see such patterns in the future (I mean you should have a rationale behind the selection of the training subset rather than looking into the test set that in reality, you do not have access to it). In such case, I call it outlier removal rather than downsampling.

Answer 4

There is a good package in python (scikit learn)

http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html

You can subsample your observations from training set using this package.