3
$\begingroup$

I am building a binary classification model using GB Classifier for imbalanced data with event rate 0.11% having sample size of 350000 records (split into 70% training & 30% testing).
I have successfully tuned hyperparameters using GridsearchCV, and have confirmed my final model for evaluation.
Results are:
Train Data-
[[244741 2] [ 234 23]]

          precision    recall  f1-score   support  

       0       1.00      1.00      1.00    244743  
       1       0.92      0.09      0.16       257  
accuracy          -         -      1.00    245000  

macro avg 0.96 0.54 0.58 245000
weighted avg 1.00 1.00 1.00 245000

test data -
[[104873 4] [ 121 2]]

          precision    recall  f1-score   support  

       0       1.00      1.00      1.00    104877  
       1       0.33      0.02      0.03       123  
accuracy          -         -      1.00    105000  

macro avg 0.67 0.51 0.52 105000
weighted avg 1.00 1.00 1.00 105000

AUC for both class 1 & 0 is 0.96
I an not sure if this is a good model I can use for predicting probability of occurrence. Please guide.
enter image description here enter image description here enter image description here

$\endgroup$
1
  • $\begingroup$ how predictable is your data? Are there well defined rules in the business model reflected in the data? How did you handle the outlier data $\endgroup$ Jul 29, 2022 at 16:39

3 Answers 3

4
$\begingroup$

"Unbalanced" data are not a problem, unless you use unsuitable error measures... like accuracy, or precision, recall and the F1 (or any other Fbeta) score, all of which suffer from exactly the same problems as accuracy. Instead, work directly with probabilistic predictions, and assess the probabilistic predictions directly using proper scoring rules.

Do not use thresholds in evaluating your statistical model. The choice of one or more (!) thresholds is an aspect of the decision, together with your probabilistic classification. It is not part of the statistical model.

We have many, many, many threads on unbalanced data at CrossValidated, and we are at a bit of a loss what to do with these, because the data science community apparently sees a problem here that completely disappears once you move away from intuitive but misleading evaluation measures. We have a Meta.CV thread dedicated to this, with a number of links to other CV threads.

$\endgroup$
1
  • $\begingroup$ I did continued with the above model and created decile summary found 100% positive events in 3rd decile for Train data and 97% in 3rd decile for test data. This looks sufficient for me as I need not care about false positives for my business case. Thank you for your valuables inputs $\endgroup$
    – RajendraW
    Jul 31, 2022 at 15:56
3
$\begingroup$

Your minority class is highly under-represented. I recommend not to proceed forward.

My suggestion would be the following:

1.) Undersample the majority class 2.) Use SMOTE to oversample the minority class 3.) Re-run the algorithm and verify the metrics

As of now, Your recall is extremely low. It signifies your model has not even identified 10% of training positive class and not even 5% of Testing positive class. This is expected considering the simplicity of your model.

I suggest you to Oversample and drop some records of the majority class. It will help you get a better view.

AUC basically only gives you Area under the curve which is based on the best tradeoff between Recall and False Positives. Since your model is highly imbalanced, AUC is not a good metric to evaluate.

$\endgroup$
3
  • $\begingroup$ Thanks, I read somewhere that it is not necessary to over/under sample data for imbalanced dataset when using ensemble tree models, hence using GBClassifier. $\endgroup$
    – RajendraW
    Jul 29, 2022 at 17:49
  • 2
    $\begingroup$ Thats wrong. Doesn't matter the model you are using. Imbalanced dataset is a universal problem and is model independent. A model is useless when not provided with proper data. It doesn't matter how powerful the architecture may be. You need to perform sampling at the base level and then proceed. You may try setting the threshold a bit low to capture more positive classes but beware this comes with the problem of increased False Positives. $\endgroup$
    – mewbie
    Jul 29, 2022 at 18:10
  • 1
    $\begingroup$ "Unbalanced" data are not a problem, and oversampling will not solve a non-problem. See also my answer with many links. $\endgroup$ Jul 30, 2022 at 7:36
3
$\begingroup$

Your model might be alright, but certainly not with a default classification threshold. As you can see, you only detected 2 out of 123 events this way. ROC_AUC tends to be overoptimistic for this level of imbalance. Studying sklearn.metrics.precision_recall_curve() could shed some light on this and help you select a decent classification threshold perhaps.

Alternatively, you may try resampling your dataset first (assuming you're mostly interested in the positive class).

$\endgroup$
4
  • $\begingroup$ Thanks for the suggestion to resample dara. have added precision recall curve in the post, can you suggest on it $\endgroup$
    – RajendraW
    Jul 29, 2022 at 17:54
  • 1
    $\begingroup$ That's a very poor curve sadly: you'll end up with either very poor recall or a disproportional amount of false positives. It looks like resampling (or reweighing, most sklearn models support class_weight='balanced' parameter) is inevitable in this case (assuming the data is good enough for reliable predictions at all). $\endgroup$
    – dx2-66
    Jul 29, 2022 at 19:17
  • $\begingroup$ Yes, I need to do this again with resampling $\endgroup$
    – RajendraW
    Jul 30, 2022 at 5:07
  • 1
    $\begingroup$ Understanding the threshold better is a good idea, but it's not part of the classification step, but of the subsequent decision step. It's better practice to clearly separate the two, for instance because there can certainly be more than two possible actions, even if there are only two classes. "Unbalanced" data are not a problem, and oversampling will not solve a non-problem. See also my answer with many links. $\endgroup$ Jul 30, 2022 at 7:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.