2
$\begingroup$

In a case of imbalanced data classification, I know that we only oversample the training set (to prevent data leakage from training to testing subsets), but what if there are no positive data points in my testing set? The testing set is still highly skewed and has only 1% of my positive class. I am using XGBoost, Random Forest, Logistic Regression, and KNN for the classification task.

Also, I have tried SMOTE, SMOTE-NC, and Class_weight to oversample my training set. To increase the chance of having more data from the minority class, I changed the 10-fold to 5-fold cross-validation (when developing the models), no improvement!

PS: I have >100K data points in my dataset.

$\endgroup$
  • 4
    $\begingroup$ Nothing stops you from oversampling your test set, but it is probably not what you want: to do a good test, you want your test set to look just like actual data. Of course you should do a stratified split of training set (validation set) and test set to make sure you have positive samples in the test set. And you need to make sure to choose an error measure suitable for imbalanced data (accuracy is the wrong choice). $\endgroup$ – Louic Aug 20 '19 at 19:01
3
$\begingroup$

Use a stratified split (as pointed out by louic in the comments). It will distribute your classes evenly across all folds. It can be done using sklearn's StratifiedKFold.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you! I was thinking about using a Hierarchical Classification (or a step-wise classification) and exclude the majority class in step #1, the second majority class in step #2 and so on. I have a 5-class classification problem. However, I am going to try StratifiedKFold to see how it works. Any thoughts on the Hierarchical idea? $\endgroup$ – Sarah Aug 20 '19 at 19:58
  • $\begingroup$ It is worth trying if just stratifying does not do the trick. $\endgroup$ – Simon Larsson Aug 20 '19 at 20:18
  • $\begingroup$ Unfortunately, stratifying did not do the trick! I am going to implement the hierarchical idea and will let you know if it helps! $\endgroup$ – Sarah Aug 20 '19 at 22:45
  • $\begingroup$ Are you sure about this part of your answer? "... It will distribute your classes evenly across all folds." I thought stratifying preserves the distribution of the original data. When I apply SMOTE on the training set (after splitting the data to train/test), train and test sets will have completely different distributions [my original dataset is HIGHLY imbalanced]. $\endgroup$ – Sarah Aug 21 '19 at 4:05
  • $\begingroup$ Yes. Let's say I have a dataset with 5 positive samples (y=1) and 1 million negative samples (y=0). If I make 5 stratified folds, then I will end up with 1 positive sample per fold. Each train set will be 4 of those folds and have 4 positive samples. Each test set will be on 1 fold and have 1 positive sample. So the imbalance does not matter, you will end up with positive samples in your test set if you do stratified folds. $\endgroup$ – Simon Larsson Aug 21 '19 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.