1
$\begingroup$

I am building a binary classification model with imbalanced target variable (13% Class 1 vs 87% class 0). I am considering the following three options to handle the data imbalance

  1. Option1: Create a balanced training dataset where with 50% / 50% split of the target variable.

  2. Option 2: Samples the dataset as-is (i.e., 87% / 13% split) and use upsampling methods (e.g., SMOTE) to balance the target variable to 50% / 50% split.

  3. Option 3: Use learning methods with appropriate hyperparameters to account for data imbalance for example: scale_pos_weight in XGBoost, class_weight in LGBMRegressor, class_weight in RandomForestClassifier

Assuming I have enough available data, is the first option is always the best approach? What are the Cons and Pros of each of the three methods? especially the 2nd and 3rd options (I assume that it is always preferred to avoiding creating new synthetic samples)

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Beside the fact that it depends also on the total size of the dataset (if the 13% is composed by thousands of instances you might simply downsample the other class), in my opinion the 3rd option is the best. But it is based on my personal experience so I won't say this is theoretically true for every scenario. $\endgroup$
    – Edoardo Guerriero
    Apr 14, 2020 at 17:52
  • $\begingroup$ @EdoardoGuerriero - Let's assume that I have a dataset with 10M samples where 1.3M is Class 1 vs 8.7M is class 0. Based on Option 1, I will, for example, train my model based on training dataset with 500K Class 1 and 500K Class 0. Are there any disadvantages of this approach compared to creating a 1M samples dataset with 870K class 0 and 130K Class 1 and then using Option 2 to balance it or option 2 to modify the weights in the learning algorithm? $\endgroup$
    – thereandhere1
    Apr 14, 2020 at 21:10
  • 1
    $\begingroup$ What I have also tried is creating multiple balanced datasets, by choosing all the samples from minority class and randomly choosing same number of records from majority class. Then create an ensemble models by training randomly on one of the datasets created. This way there is less loss of information and models have a representation of the whole data. $\endgroup$
    – nishant
    May 17, 2020 at 4:59
  • 1
    $\begingroup$ Always consider also option 0: don't do anything. In probabilistic classifiers, often (usually?) no action is required, provided you use those probabilities (not just making hard classifications at threshold 0.5, e.g.). $\endgroup$
    – Ben Reiniger ♦
    May 18, 2020 at 0:41
  • 1
    $\begingroup$ Complete agreement with @BenReiniger Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? $\endgroup$
    – Dave
    Nov 2, 2022 at 13:20

1 Answer 1

Reset to default
0
$\begingroup$

I think it mostly depends on your dataset type! are you dealing with text? or image? or... and your features will tell which option is the best fit for your case....but according to my experience in most of the cases, option 1 and 2 besides they depend on your dataset and power of your features they need to be judge based on your model high bias or variance and they should inform you they are good or no! you need to do some experiment to figure out them or know your dataset well to find out adding or reducing dataset will affect your model performance or not!

and what I like to tell is try to use upsampling and downsampling methods same time to make your dataset balanced in a fair way(kinda)!....in this case (87% class 0 and 13% class 1)....upsample class 1 and downsample class 0! how much you need to upsample or how much downsample it is all your choice and definition of fairness in your dataset! and this definition could differ!

Improve this answer
$\endgroup$
1
  • $\begingroup$ I am building an ad response model. Would not the most 'honest' approach is to simply create a dataset which is already balanced without the need for synthetic samples, replicating samples, or weight modifications in the learning algorithm? How will the features affect the appropriate approach to be used? $\endgroup$
    – thereandhere1
    Apr 14, 2020 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.