11
$\begingroup$

I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of the training set and testing set are similar.
And I search it and find a blog which check the similarity of distributions by converting it into a binary classification problem. If it gets a high AUC, the distribution of the training set and the testing set must be different. And the idea he gives as follows:

If there exists a covariate shift, then upon mixing train and test we’ll still be able to classify the origin of each data point (whether it is from test or train) with good accuracy.

But I still can't understand why he can check the similarity of these two distributions in this way.
And are there other ways to check the similarity of it?
It will be appreciated, if anyone could help me.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
8
$\begingroup$

It looks like the person that wrote the blog is combining the samples from the test set and train set into one dataframe and then predicting if each sample comes from the test set or training set (his y variable is called “is_train” which indicates whether the sample came from the training set or not). I think his point is that if you are able to accurately classify whether the sample comes from the test or training set then the predictor variables have different underlying distributions. That would signify that your original model will probably not work well on this test data. Also — he is using AUROC as the performance metric. A high AUROC means that the model is performing well, and in this case it means that there is a big difference in distributions of predictor variables between the training and test set. Ideally, the distribution of the predictors for the training and test set should be the same, so you would want to get an AUROC that is close to 0.5.

I think this situation would only be relevant in cases where you have your model deployed and you need to check if your model is still relevant over time. If you are building a new model you shouldn’t need to do something like this because the test data is randomly sampled from the dataset. Additionally, if you’re doing cross validation then there is even less reason to worry about something like that.

Improve this answer
$\endgroup$
3
$\begingroup$

Two common scores to quantify the (dis)similarity of distributions are the Kullback-Leibler divergence and the Jensen-Shannon divergence.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thats true, but you would have to compare variables individually from test and training sets. $\endgroup$
    – fractalnature
    Commented Apr 18, 2019 at 15:17
  • $\begingroup$ Using the t-SNE approach, you should be able to minimize the sum of Kullback-Leibler divergence of overall data points using a gradient descent method. $\endgroup$
    – Sarah
    Commented Oct 16, 2019 at 0:12
  • $\begingroup$ @Sarah do you know a source that performs multivariate KLD minimization? $\endgroup$
    – develarist
    Commented Sep 19, 2020 at 6:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.