2
$\begingroup$

I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a workflow in mind which I wanted to share and get an opinion on if I'm heading in the right direction or maybe I missed something.

  1. Split Train/Test/Val
  2. Setup pipeline for GridSearch and optimize hyper-parameters (pipeline will only oversample training folds)
  3. Scoring metric will be AUC as training set is balanced at that point
  4. Since model was trained on balanced dataset, it will probably be very conservative and predict a lot of false positives
  5. Taking above into consideration, model will be calibrated to have more accurate probabilities (CalibratedClassifierCV)
  6. View precision/recall curve with calibrated probability thresholds on validation set and determine optimal point

Does this process sound reasonable? Would appreciate any feedback/suggestions

Improve this question
$\endgroup$
2
  • $\begingroup$ I've been pulling together a similar workflow for making predictions using imbalanced classes and your steps seem thorough to me. One thing I'm noticing in my current project is that the calibration of predict_proba thresholds seems to significantly reduce or even erase the benefit of resampling for many of my classifiers. After adjusting the thresholds for models fit to raw (imbalanced) data, they achieve scores (f1, precision, recall, Cohen's kappa) comparable to models fit to imblearn-resampled data that have also been re-thresholded. $\endgroup$
    – David Diaz
    Mar 18, 2020 at 15:37
  • $\begingroup$ Are you sure that the imbalance is a problem in need of addressing? $\endgroup$
    – Dave
    Jan 4 at 4:49

1 Answer 1

Reset to default
0
$\begingroup$

I am not sure if in the last point, you meant the validation set instead of the testing set.

Here is my advice: 1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need to handle the data imbalance problem.

2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.

3- use the training set for training, the validation set for tuning the hyper parameters , BUT do not touch the testing set

4- use the testing set for testing only

5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.