1
$\begingroup$

I am building a binary classification model with imbalanced target variable (13% Class 1 vs 87% class 0). I am considering the following three options to handle the data imbalance

  1. Option1: Create a balanced training dataset where with 50% / 50% split of the target variable.
  2. Option 2: Samples the dataset as-is (i.e., 87% / 13% split) and use upsampling methods (e.g., SMOTE) to balance the target variable to 50% / 50% split.

  3. Option 3: Use learning methods with appropriate hyperparameters to account for data imbalance for example: scale_pos_weight in XGBoost, class_weight in LGBMRegressor, class_weight in RandomForestClassifier

Assuming I have enough available data, is the first option is always the best approach? What are the Cons and Pros of each of the three methods? especially the 2nd and 3rd options (I assume that it is always preferred to avoiding creating new synthetic samples)

$\endgroup$
6
  • 1
    $\begingroup$ Beside the fact that it depends also on the total size of the dataset (if the 13% is composed by thousands of instances you might simply downsample the other class), in my opinion the 3rd option is the best. But it is based on my personal experience so I won't say this is theoretically true for every scenario. $\endgroup$ Apr 14, 2020 at 17:52
  • $\begingroup$ @EdoardoGuerriero - Let's assume that I have a dataset with 10M samples where 1.3M is Class 1 vs 8.7M is class 0. Based on Option 1, I will, for example, train my model based on training dataset with 500K Class 1 and 500K Class 0. Are there any disadvantages of this approach compared to creating a 1M samples dataset with 870K class 0 and 130K Class 1 and then using Option 2 to balance it or option 2 to modify the weights in the learning algorithm? $\endgroup$ Apr 14, 2020 at 21:10
  • 1
    $\begingroup$ What I have also tried is creating multiple balanced datasets, by choosing all the samples from minority class and randomly choosing same number of records from majority class. Then create an ensemble models by training randomly on one of the datasets created. This way there is less loss of information and models have a representation of the whole data. $\endgroup$
    – nishant
    May 17, 2020 at 4:59
  • 1
    $\begingroup$ Always consider also option 0: don't do anything. In probabilistic classifiers, often (usually?) no action is required, provided you use those probabilities (not just making hard classifications at threshold 0.5, e.g.). $\endgroup$
    – Ben Reiniger
    May 18, 2020 at 0:41
  • 1
    $\begingroup$ Complete agreement with @BenReiniger Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? $\endgroup$
    – Dave
    Nov 2, 2022 at 13:20

2 Answers 2

0
$\begingroup$

I think it mostly depends on your dataset type! are you dealing with text? or image? or... and your features will tell which option is the best fit for your case....but according to my experience in most of the cases, option 1 and 2 besides they depend on your dataset and power of your features they need to be judge based on your model high bias or variance and they should inform you they are good or no! you need to do some experiment to figure out them or know your dataset well to find out adding or reducing dataset will affect your model performance or not!

and what I like to tell is try to use upsampling and downsampling methods same time to make your dataset balanced in a fair way(kinda)!....in this case (87% class 0 and 13% class 1)....upsample class 1 and downsample class 0! how much you need to upsample or how much downsample it is all your choice and definition of fairness in your dataset! and this definition could differ!

$\endgroup$
1
  • $\begingroup$ I am building an ad response model. Would not the most 'honest' approach is to simply create a dataset which is already balanced without the need for synthetic samples, replicating samples, or weight modifications in the learning algorithm? How will the features affect the appropriate approach to be used? $\endgroup$ Apr 14, 2020 at 21:02
0
$\begingroup$

I think it depends on the size of your data and the predictive power of your features, what your reference optimization metric is and data quality.

Using class weights or resampling would increase your model's performance in predicting the positive/minority classes but might also bring extra noise - false positives. On top of that, you do not always have to make a perfectly balanced 50:50 dataset. For example for me, I was dealing with severe imbalance and without class weights or SMOTE, the performance of my model was really never good enough.

However, one disadvantage of your Options 2-3 is that your predictive probabilities will be inflated depending on how much you apply them. So if you are after predicted probabilities (which can be the case in marketing use cases) then you might have to make a compromise between how much you want to improve your model's performance vs how much you want to sacrifice the validity of your probabilities.

Lastly, you can also try Logistic Regression if it works as well as others, it is computationally much simpler and you can interpret your coefficients more easily too to get business insights.

New contributor
Koray Can Canut is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.