I am applying both class_weight and oversampling (SMOTE) techniques on a multiclass classification problem and getting better results when using the class_weight technique. Could someone please explain what could be the cause of this difference?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ You should check out this question: datascience.stackexchange.com/questions/44755/… $\endgroup$– Mark.FMay 26, 2019 at 5:36
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
You should not expect class_weight parameters and SMOTE to give the exact same results because they are different methods.
Class weights directly modify the loss function by giving more (or less) penalty to the classes with more (or less) weight. In effect, one is basically sacrificing some ability to predict the lower weight class (the majority class for unbalanced datasets) by purposely biasing the model to favor more accurate predictions of the higher weighted class (the minority class).
Oversampling and undersampling methods essentially give more weight to particular classes as well (duplicating observations duplicates the penalty for those particular observations, giving them more influence in the model fit), but due to data splitting that typically takes place in training this will yield slightly different results as well.
SMOTE creates new observations of the minority class by randomly sampling from a set of "similar" minority class observations. Synthesized observations are computed based off adding a random percentage of the difference between two randomly chosen "similar" observations, for each coordinate (i.e. column). Similar observations are defined typically using the k most closest neighbours to a particular observation of the minority class. This means that, depending on the value of k chosen, as well as numerous other factors such as how similar your observations are in general, the distance measure, etc. SMOTE may or may not be useful for your particular problem.
Methods that deal with class imbalance do not all work the same and is a large area of study. It appears you have noticed that the class weight method is more effective, which is why any sort of method chosen in addressing any potential class imbalance problem needs to be wrapped within a model validation scheme to see if the particular method (or really, using any method at all) yields any benefit.
-
1$\begingroup$ Thank you for the detailed explanation. I have a follow-up question: when I compare the iba (Index of Balanced Accuracy) index of these two techniques, SMOTE outperforms class_weight. The recall of the minority class in class_weight is much less than the SMOTE approach. I thought we may need to involve iba in our decision-making process. May I have your thoughts on this? $\endgroup$– SarahMay 26, 2019 at 23:19
-
$\begingroup$ I am not too familiar with iba because I almost never use evaluation metrics that aren't proper scoring rules. Corrections can be made to metrics such as log loss (such as finding the relative improvement of the model's log loss vs the log loss of a simple, naive model such as one that always predicts the majority class with probability equal to the proportion in the training set). That being said, iba appears to be the average accuracy score observed within each class? If you value better predictions of the minority class (more true positives) at the expense of more false positives then... $\endgroup$– aranglolMay 27, 2019 at 0:49
-
$\begingroup$ SMOTE might be the better option based off what you have observed. But this depends on the context of what you are trying to do, namely the costs of false positives/false negatives. Rarely does anything come for free. In your case, the model's overall accuracy is dropping (I assume so, anyway) in exchange for predicting the minority class better. Seemingly, this is because your model is more likely to predict the minority class through the use of SMOTE, for better (more true positives) and for worse (more false positives). Whether this is desirable depends solely on the context of how... $\endgroup$– aranglolMay 27, 2019 at 0:49
-
1
$\begingroup$
$\endgroup$
1
Probably not the answer you're looking for, but don't go crazy! Different class weight strategies give different results.
The follwing drove me almost crazy! The following should give the same results, but it doesn't.
class_weight = "balanced"
class_weight={0:0.85, 1:0.15}
I learned to live with it ...
-
1$\begingroup$ Thank you @FrancoSwiss, the example is very intuitive. $\endgroup$– SarahMay 26, 2019 at 21:45