I have a binary classification problem with imbalanced data and am attempting to use cost-sensitive learning to handle the imbalance. I have used LogisticRegression, LinearSVC, SVC and DecisionTree algorithms and am tuning the class weights using GridSearch to achieve the optimal F score, like so:

from sklearn.model_selection import GridSearchCV, StratifiedKFold
model = DecisionTreeClassifier()
#Setting the range for class weights
weights = np.linspace(0.0,0.99,200)

#Creating a dictionary grid for grid search
param_grid = {'class_weight': [{0:x, 1:1.0-x} for x in weights]}

#Fitting grid search to the train data with 5 folds
gridsearch_dt = GridSearchCV(estimator= model, 
                          param_grid= param_grid,
                          verbose=2).fit(X_train, y_train)

After tuning the class_weight parameter for each of the four algorithms, I then plotted the results like so:enter image description here

I then complied all the results in a table:

enter image description here

My understanding is that a larger weight is assigned to the class with more importance, and a smaller weight is assigned to a class with less importance. In the case of imbalanced data, I would expect all models to assign higher weights to the minority class, as is the case in the LR, SVC and Linear SVC model. However, only DT assigned a higher weight to the majority class to achieve an optimal F score, and I am trying to understand this behaviour.

I have been searching through past literature to understand the root cause of this and I came across this article:

We have shown that commonly used decision tree splitting criteria are relatively insensitive to cost. That in fact, a newly introduced criterion is completely cost insensitive.

I am trying to make sense of all of this but need some help! Does this article explain DT's behaviour with cost-sensitive learning?

I appreciate some help, thank you!



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy