2
$\begingroup$

Consider a binary classification scenario whereby the True class (5%) is severely outbalanced to the False class (95%). My data set contains numeric data. I am using SKLearn and trying some different algorithms such as Gradient Boosting Classifier (GCB), Random Forest (RDC) and Support Vector Classifier (SVC). Due to the unbalanced aspect, I am using "sample_weight" in all the methods (fit, score, confusion_matrix, etc) and populating it with the below weight array, whereby, True values are given a value of 20 and False values are given a value of 1.

sample_weight = np.array([20 if i == 1 else 1 for i in y_test])

This is supposed to "balance" the classification. Is this a correct approach first of all? And what are the implications? Increasing the sample size with real data is impossible. Overfitting would be another potential option, but I thought to start by the "sample_weight" approach due to the significant discrepancy between classes.

Now consider these results:

GCB - Accuracy 71%, Precision 74%, Recall 69%, F1-Score 71%

RFC - Accuracy 67%, Precision 82%, Recall 47%, F1-Score 60%

SiVC - Accuracy 63%, Precision 74%, Recall 45%, F1-Score 56%

GCB outperforms the other algorithms in Accuracy. RFC in Precision. Recall is very bad in both RFC and SVC, and the least value in the 3 cases.

My next question is mainly with regards to recall. The False negatives are considerable - as shown by the recall (in the SVC case, 55% of actual True values were predicted as False); is this because in the original data set there are more False values?

Where does the sample weighting come in in all of this? Isn't the sample weighting supposed to mitigate? Or does the class imbalance still affect the classification regardless of the weighting; in some cases more than others (case in point RFC and SVC more than GCB)?

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ Welcome to Cross Validated! It would help if you said why you find imbalance to be a problem. Statisticians do not see class imbalance as such as issue. stats.stackexchange.com/questions/357466 fharrell.com/post/class-damage fharrell.com/post/classification stats.stackexchange.com/a/359936/247274 stats.stackexchange.com/questions/464636 twitter.com/f2harrell/status/1062424969366462473?lang=en $\endgroup$
    – Dave
    Commented Jan 14, 2022 at 1:16
  • 1
    $\begingroup$ Hi @Dave, thank you for your comment. The mentioned binary classifiers (and most other binary classifiers - to my knowledge) would be biased towards the majority class, in cases of imbalance. In my case, if the classifier predicts all rows as False, it would be 95% accurate, which obviously wouldn't be right. I want to eliminate bias. I want the classifier to value True predictions, just as much as the False ones, albeit the sample size difference. $\endgroup$
    – Jurgen Cuschieri
    Commented Jan 14, 2022 at 1:27
  • 1
    $\begingroup$ Most binary classifiers (e.g., logistic regressions and neural networks) don’t even predict categories at all. Did you read any of the links I posted? $\endgroup$
    – Dave
    Commented Jan 14, 2022 at 1:34
  • 1
    $\begingroup$ I am still going through them one by one.. quite a lot to read and make sense of :). I replied the moment I saw your comment to let you know my reasoning and the reason I posted in the first place. I will let you know whether the links you posted put me into the right direction $\endgroup$
    – Jurgen Cuschieri
    Commented Jan 14, 2022 at 1:39
  • $\begingroup$ Hi @Dave, I found them quite interesting, especially the first one (and the many sub links I ended up into via that post). I am still not sure of the way forward though. But have definitely absorbed some valuable information. In the meantime I retried the GCB without the weighting and sure enough I got: 95% accuracy, 49% precision, 3% recall, 6% F1-score. That doesnt look too great. It could also be that my sample size is too small. In any case I will check the others too. $\endgroup$
    – Jurgen Cuschieri
    Commented Jan 14, 2022 at 2:44

1 Answer 1

Reset to default
3
$\begingroup$

Remember, that in higly imbalanced data model does not learn anything as it minimises its objective function just by predicting everything to majority class. Yes Sample weight values you have assigned seems to do the right thing. What sample weight does is tweak the objective function to consider one error in predicting True class equivalent to 20 error in Negative class. This forces model to learn as it cannot minimize its objective function just by predicting majority class. this is how sample weights play a part in imbalanced class.

Sample weight is not a panacea it has improved model performance a lot but may not give you the best solution. To improve recall you should try following :

  1. Creating a more complex model by increasing depth, iteration etc depending on model class.

  2. Sample weight can also by a hyperparamets which can be tunes

Just one suggestion if you are giving same sample weights to all observation as same you could have done using class weights also.

Improve this answer
$\endgroup$
2
  • $\begingroup$ hi, thank you for your answer. Can you clarify on point 2, and the last sentence? All is clear apart from those 2 remarks. $\endgroup$
    – Jurgen Cuschieri
    Commented Jan 16, 2022 at 16:25
  • $\begingroup$ Yes, You can try multiple sample weights so you can tune sample weights as hyperparameters. Also as you are giving sample weights similar for all observations you can directly use class weights $\endgroup$
    – Ashwiniku918
    Commented Jan 16, 2022 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.