# Imbalanced data set with Sample weighting - How to interpret the performance metrics?

Consider a binary classification scenario whereby the True class (5%) is severely outbalanced to the False class (95%). My data set contains numeric data. I am using SKLearn and trying some different algorithms such as Gradient Boosting Classifier (GCB), Random Forest (RDC) and Support Vector Classifier (SVC). Due to the unbalanced aspect, I am using "sample_weight" in all the methods (fit, score, confusion_matrix, etc) and populating it with the below weight array, whereby, True values are given a value of 20 and False values are given a value of 1.

sample_weight = np.array([20 if i == 1 else 1 for i in y_test])


This is supposed to "balance" the classification. Is this a correct approach first of all? And what are the implications? Increasing the sample size with real data is impossible. Overfitting would be another potential option, but I thought to start by the "sample_weight" approach due to the significant discrepancy between classes.

Now consider these results:

GCB - Accuracy 71%, Precision 74%, Recall 69%, F1-Score 71%

RFC - Accuracy 67%, Precision 82%, Recall 47%, F1-Score 60%

SiVC - Accuracy 63%, Precision 74%, Recall 45%, F1-Score 56%

GCB outperforms the other algorithms in Accuracy. RFC in Precision. Recall is very bad in both RFC and SVC, and the least value in the 3 cases.

My next question is mainly with regards to recall. The False negatives are considerable - as shown by the recall (in the SVC case, 55% of actual True values were predicted as False); is this because in the original data set there are more False values?

Where does the sample weighting come in in all of this? Isn't the sample weighting supposed to mitigate? Or does the class imbalance still affect the classification regardless of the weighting; in some cases more than others (case in point RFC and SVC more than GCB)?

• Welcome to Cross Validated! It would help if you said why you find imbalance to be a problem. Statisticians do not see class imbalance as such as issue. stats.stackexchange.com/questions/357466 fharrell.com/post/class-damage fharrell.com/post/classification stats.stackexchange.com/a/359936/247274 stats.stackexchange.com/questions/464636 twitter.com/f2harrell/status/1062424969366462473?lang=en
– Dave
Jan 14 at 1:16
• Hi @Dave, thank you for your comment. The mentioned binary classifiers (and most other binary classifiers - to my knowledge) would be biased towards the majority class, in cases of imbalance. In my case, if the classifier predicts all rows as False, it would be 95% accurate, which obviously wouldn't be right. I want to eliminate bias. I want the classifier to value True predictions, just as much as the False ones, albeit the sample size difference. Jan 14 at 1:27
• Most binary classifiers (e.g., logistic regressions and neural networks) don’t even predict categories at all. Did you read any of the links I posted?
– Dave
Jan 14 at 1:34
• I am still going through them one by one.. quite a lot to read and make sense of :). I replied the moment I saw your comment to let you know my reasoning and the reason I posted in the first place. I will let you know whether the links you posted put me into the right direction Jan 14 at 1:39
• Hi @Dave, I found them quite interesting, especially the first one (and the many sub links I ended up into via that post). I am still not sure of the way forward though. But have definitely absorbed some valuable information. In the meantime I retried the GCB without the weighting and sure enough I got: 95% accuracy, 49% precision, 3% recall, 6% F1-score. That doesnt look too great. It could also be that my sample size is too small. In any case I will check the others too. Jan 14 at 2:44