I have used the 'class_weight' method to balance my multi-class classification problem, using Logistic Regression, Random Forest, and XGBoost classifiers. Among these three methods, logistic regression's performance for the minority classes is substantially higher than the other two models. Could someone please explain why does LR beat decision-tree-based classifiers in this scenario? Thank you.
Very good question! I think the plain answer is: no. To my experience this usually depends on the data. Even different boosting methods/tools will perform differently. It often depends on try and error.
What can make a difference is using L1/L2 regularization, since it can shrink features which are „not useful“ to make a good prediction.
I cannot think of a theoretical reason here. I‘m curious if someone comes up with a more theory based answer!
Seems I found some reasonable justifications for this behavior!
Class_weight in the LogisticRegression approach is applied to sample_weight, which is used in a few internal functions like >_logistic_loss_and_grad, _logistic_loss, etc.:
#Logistic loss is the negative of the log of the logistic function..
out = -np.sum(sample_weight * log_logistic(yz)) + .5 * alpha * np.dot(w, w)
NOTE: ---> ^^^^^^^^^^^^^
Likewise, in decision-tree based approaches like RandomForest and XGBoosting, the class_weigh is applied to the
entropy function, which impacts both nominator and denominator --> less impact on the purity function!