2
$\begingroup$

In a case of imbalanced data classification, I know that we only oversample the training set (to prevent data leakage from training to testing subsets), but what if there are no positive data points in my testing set? The testing set is still highly skewed and has only 1% of my positive class. I am using XGBoost, Random Forest, Logistic Regression, and KNN for the classification task.

Also, I have tried SMOTE, SMOTE-NC, and Class_weight to oversample my training set. To increase the chance of having more data from the minority class, I changed the 10-fold to 5-fold cross-validation (when developing the models), no improvement!

PS: I have >100K data points in my dataset.

Improve this question
$\endgroup$
1
  • 4
    $\begingroup$ Nothing stops you from oversampling your test set, but it is probably not what you want: to do a good test, you want your test set to look just like actual data. Of course you should do a stratified split of training set (validation set) and test set to make sure you have positive samples in the test set. And you need to make sure to choose an error measure suitable for imbalanced data (accuracy is the wrong choice). $\endgroup$
    – Louic
    Commented Aug 20, 2019 at 19:01

1 Answer 1

Reset to default
3
$\begingroup$

Use a stratified split (as pointed out by louic in the comments). It will distribute your classes evenly across all folds. It can be done using sklearn's StratifiedKFold.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you! I was thinking about using a Hierarchical Classification (or a step-wise classification) and exclude the majority class in step #1, the second majority class in step #2 and so on. I have a 5-class classification problem. However, I am going to try StratifiedKFold to see how it works. Any thoughts on the Hierarchical idea? $\endgroup$
    – Sarah
    Commented Aug 20, 2019 at 19:58
  • $\begingroup$ It is worth trying if just stratifying does not do the trick. $\endgroup$
    – Simon Larsson
    Commented Aug 20, 2019 at 20:18
  • $\begingroup$ Unfortunately, stratifying did not do the trick! I am going to implement the hierarchical idea and will let you know if it helps! $\endgroup$
    – Sarah
    Commented Aug 20, 2019 at 22:45
  • $\begingroup$ Are you sure about this part of your answer? "... It will distribute your classes evenly across all folds." I thought stratifying preserves the distribution of the original data. When I apply SMOTE on the training set (after splitting the data to train/test), train and test sets will have completely different distributions [my original dataset is HIGHLY imbalanced]. $\endgroup$
    – Sarah
    Commented Aug 21, 2019 at 4:05
  • $\begingroup$ Yes. Let's say I have a dataset with 5 positive samples (y=1) and 1 million negative samples (y=0). If I make 5 stratified folds, then I will end up with 1 positive sample per fold. Each train set will be 4 of those folds and have 4 positive samples. Each test set will be on 1 fold and have 1 positive sample. So the imbalance does not matter, you will end up with positive samples in your test set if you do stratified folds. $\endgroup$
    – Simon Larsson
    Commented Aug 21, 2019 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.