1
$\begingroup$

I want to understand how to split the imbalanced data set with a binary target variable where 87% of the samples are negative and 13% of the samples are positive. Now, I know that you should always split the data into train and test set before doing any kind of resampling to avoid info leakage but how what strategy gives better results - random train_test_split or StratifiedShuffleSplit? The general sentiment is that the latter seems to be outperforming the former in case of an imbalanced data set. Btw, how does StratifiedShuffleSplit work?

Also, how to factor in the cross-validation set and how to process it?

Should it also contain the same number of positive and negative samples as per the sampling was done on the training set?

| improve this question | |
$\endgroup$
2
$\begingroup$

Grzegorz's answer gets to the root of the problem - if the future data this model will be used to make predictions on will have the same distribution, then stratification by class % makes sense.

One thing I wanted to add is I typically use the normal train_test_split function and just pass the class labels to its stratify parameter like so:

train_test_split(X, y, random_state=0, stratify=y, shuffle=True)

This will both shuffle the dataset and match the %s of classes in the result of train_test_split.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Got it. How do you then resample data? Do you oversample or undersample the validation set too or just the training set and let the class distribution in the validation as the holdout set? $\endgroup$ – Krishnang K Dalal Nov 1 '19 at 9:01
  • 1
    $\begingroup$ I only apply the oversampling/undersampling to the training set, and not validation. One way to do this conveniently is using samplers from the imblearn package. They have their own Pipeline object that won't apply the transformation to validation. Some more info is here: stackoverflow.com/questions/50245684/… $\endgroup$ – Marc Kelechava Nov 1 '19 at 17:25
  • $\begingroup$ That link is just wonderful! Thanks a lot. So, I believe I can Stratify split my dataset into the training and the holdout set and use GridSearchCV with the Pipeline from imblearn on the training set. Does that sound like a good approach? $\endgroup$ – Krishnang K Dalal Nov 2 '19 at 17:46
  • 1
    $\begingroup$ No problem, and that approach should work well. Also not sure if this will be helpful for you, but this paper (particularly pg. 8 - pg. 10) helped me a lot in thinking about what metrics to consider when evaluating imbalanced datasets: arxiv.org/pdf/1810.07168v1.pdf $\endgroup$ – Marc Kelechava Nov 2 '19 at 21:18
  • $\begingroup$ Thanks a lot, I'll check it out! $\endgroup$ – Krishnang K Dalal Nov 4 '19 at 7:16
0
$\begingroup$

It all depends on what is your objective. For instance, if you deploy a model where you expect that the test data (e.g. streaming data) will have the same pos/neg distribution then it makes sense to stratify your splitting. In that way, you bias your learning procedure to label more as negative. On the other hand, if you are told that more weight is given to detect the sparse positive examples then you may want to try an equal distribution.

| improve this answer | |
$\endgroup$
0
$\begingroup$

For a large dataset, non-stratified splitting is usually okay. You have to consider what is the probability of significant differences in the balance between test and training sets. For me, 13% of thousands of records is probably no issue but 13% of a hundred records is likely to be an issue.

Marc's answer explains how the stratification works. StratifiedShuffleSplit just makes this easier in the context of cross-validation by creating a generator to split it multiple times. You can also pass it the random_state so that the splits remain consistent in each fold across iterations, which is important for hyper-parameter tuning. Check out the example in the documentation.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.