Why does class_weight usually outperform SMOTE?

Question

I'm trying to figure out what exactly class_weight from sklearn does.

When working with imbalanced datasets, I'm always using class_weight because the results are usually better than using SMOTE. However, I'm not sure why.

I've tried to find an answer, but most of answers regarding the subject are vague. For instance, the first answer here explain class_weight in a way that looks similar to SMOTE. This and this also didn't provide an answer.

I read once that SMOTE is used as an oversampling method that relies on KNN and that class_weight acts on the cost function. But I didn't find this anymore and I'm not sure it's true, since I haven't read anywhere else.

Answer 1

The difference is class_weight does not synthesize new data, but SMOTE does.

If your model results always favor class_weight, then we can ask whether your SMOTE is synthesizing good samples. One way to see what your SMOTE engine is doing can be - what labels would your SMOTE assign to your test data? How many % for assigned label = true label? One reason for poor kNN-based SMOTE is that kNN is a distance based method and it can mess up when your data has a high dimensionality (too many features). For example, you can have 2 samples having the same value for 99 out of the 100 features but a difference of 1000 in the last feature, and you can have another pair of samples having a difference of 10 in each of the 100 features. The final euclidean distance for the former case is 1000, but the later case would be just 100. If you think that the former pair should be "closer", the distance metrics says the opposite.

So I would say SMOTE literally requires your understanding in the features to select the right configuration, such as what distance metrics to use, and perhaps what features to consider.

Answer 2

One possible reason is the scikit-learn's class_weight parameter can potentially give different weight values to different instances based on category membership. From class_weight documentation:

If ‘balanced’, class weights will be given by n_samples / (n_classes * np.bincount(y)). If a dictionary is given, keys are classes and values are corresponding class weights. If None is given, the class weights will be uniform.

SMOTE stands for Synthetic Minority Over-sampling Technique. The synthetic part means SMOTE fabricates novel samples based on existing data. SMOTE does this by interpolating new instances based on existing feature values.

In the cases when class_weight outperforms SMOTE, increasing the value of existing data is more useful than generating new data.