# Different Test Set and Training Set Distribution

I am working on a data science competition for which the distribution of my test set is different from the training set. I want to subsample observations from training set which closely resembles test set.

How can I do this?

• Random Under-Sampling, Random Over-Sampling,Cluster-Based Over Sampling,Informed Over Sampling: Synthetic Minority Over-sampling Technique,Modified synthetic minority oversampling technique (MSMOTE) etc.. Commented Feb 28, 2018 at 7:40

Great question, this is what is known in Machine Learning paradigm as either "Covariate Shift", or "Model Drift" or "Nonstationarity" and so on.

One of the critical assumption one would make to build a machine learning model for future prediction is that unseen data (test) comes from the same distribution as training data! However, in reality this rather simple assumption breaks easily and upcoming data (its distribution) changes over time for many reasons. For those who may not be familiar with this very important problem, I encourage looking here or post!

To me, your question falls into the same category. Although I do not have the perfect solution (an implementation to offer), but I think you may look:

• This blog post gives you a simple way to handle the subsampling of training data with code provided in Python!
• Check this research paper. They propose to solve the problem by reweighting the training data so that the distribution of training is closer to the distribution of test using Kullback-Leibler Importance Estimation Procedure base on "Kullback-Leibler divergence" theorem. I do not know if they provide an implementation or it can be implemented easily, but I think it might worth digging as it sounds a professional way to deal the distribution mismatch.

QUICK update (a good solution): I found a Python implementation of KLIEP algorithm of that research paper (last point) to find those weights. It rather seems easy to use! Basically it resamples the training by putting weights (via the KLIEP algorithm) so that the assumption of having a similar distribution of train and test holds true as much as possible.

• By subsampling training data based on the test data, would we be essentially leaking information from test data to the training data? (as pointed by @Arnaud in their answer) EDIT: Say we have subsampled training data, and the model performs well in the test data too. After implementation, real world data may come from another unseen distribution, there is no guarantee that it will follow test data, since test data itself doesn't follow training data. Commented Jun 29, 2021 at 8:47
• Did you read the first blog-post about subsampling? What leaking? You are leaking data into test set, you are subsampling to make the dist. of train features closer to test. This is a let's say a method to automatically make a well-generalized model more specific and better performing on an unseen set of data that differ largely from train set. Also sure, there is no guarantee that holds true, that is why this has be automated and be checked every now and then depending on the nature of the data and business. Commented Jun 30, 2021 at 6:47
• I disagree also with this. Assume we have dataset X and we divide into datasets Z and G. The distributions are assumed when we divide the dataset into two. Let's assume that G is our test dataset and we will fit Z's distribution into G's. But what makes G the "truer" distribution than "Z's". Especially that, Z is usually the bigger one (usually training dataset is bigger). So the purpose of the resampling here is "to improve results on the test dataset" which is basically "data leaking" or kind of cheating. Commented Aug 7, 2021 at 9:05
• Also, the paper is in a very bad, non-statistics non-ML oriented Journal with low impact factor. Which would make it suspicious. So @Gautam J is right from my point of view. Commented Aug 7, 2021 at 9:06
• [2/2] If we have complete test data (both predictor and target variables), then we could just train our model on the test data. This subsampling / "covariate shift technique" can be thought of as a half-measure, which we may choose when the test set includes predictors, but not targets. Alternatively, we could think of this as a cheap and simplified variety of semi-supervised learning. DanielWiczew and @GautamJ 's comments have their merits, but this does not seem to be the full story. Commented Aug 24, 2022 at 2:54

I want to subsample observations from training set which closely resembles test set.

I am not sure you'd want to do that. The whole purpose is rather to train your algorithm so that it generalises well to unseen data.

Usually, one should adapt its test data to its train data (e.g. standardising test data according to train data) and not the other way around. In practice, you don't know your test data.

Train set subsampling might not be the best solution!

The differences between test/execution set and training set distribution/features are very common in supervised learning tasks (this is one of the reasons that competitions such as Kaggle are challenging). That is why we say the past performance may be (only) used as a guide for estimating the future performance but it does not indicate/guarantee it. Therefore, generalizable models have always been preferred over fine-tuned models that may perform very well on the train (sub)set but do poorly on unseen data.

While such difference is normal, the too large gap between the past and future sample may be referred as examples of concept drift which is an active research field by itself. Given your question, I cannot judge that your case is a normal ML case or the concept drift is really happening.

These are my suggestions:

1. Train a number of models with high generalization capability. Using bootstrap sampling from your train dataset, you can easily calculate bias and variance components of errors. Recall that you are looking for a low-variance model (where the changes in data would have a marginal effect on its performance) rather than low-bias but high-variance models (that might overfit to your training (sub)set). Now, you can select the best algorithms and evaluate them against the test set. Note that in the training time we supposed to not look at the test set!

2. Instead of several random downsampling, look for standardization/normalization and feature selection/engineering. These techniques might be practical in learning more general models. For example, sometimes the range of feature domain may change over time while the shape of distribution (whatever it is) remains almost the same (eg same distribution that is shifted towards left or right). In such case, a simple standardization (ie mapping the train and test samples to a predefined space such as [0,1] using different mapping functions) can reduce the symptoms.

3. Systematic downsampling can only be an appropriate solution if you do it based on some knowledge about the problem (not just for the purpose of getting a better accuracy on the test dataset). For example, you might know that some of the records in the train data are sampled a long time ago, from far field, or affected by particular factors which none of them will happen in future (in test data collection). In such case, you may remove those samples that can be irrelevant as you are confident that you will not see such patterns in the future (I mean you should have a rationale behind the selection of the training subset rather than looking into the test set that in reality, you do not have access to it). In such case, I call it outlier removal rather than downsampling.

• Dataset Shift in Machine Learning is a good, rigorous overview of the field. "Covariate (input) shift means that only the input distribution changes, whereas the conditional distribution of the outputs given the inputs $p(y|x)$ remains unchanged." Commented Oct 10, 2018 at 18:13

There is a good package in python (scikit learn)

http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html

You can subsample your observations from training set using this package.

• As far as I understand the question, train/test distributions are different which if not taken into account would lead to what is known "Covariate Shift". Simple subsample using "train_test_split" implementation in scikit learn mentioned here won't take into account distributions during split! Thus the answer is not relevant. Commented Feb 27, 2018 at 9:28