3
$\begingroup$

I have used the 'class_weight' method to balance my multi-class classification problem, using Logistic Regression, Random Forest, and XGBoost classifiers. Among these three methods, logistic regression's performance for the minority classes is substantially higher than the other two models. Could someone please explain why does LR beat decision-tree-based classifiers in this scenario? Thank you.

$\endgroup$
3
$\begingroup$

Very good question! I think the plain answer is: no. To my experience this usually depends on the data. Even different boosting methods/tools will perform differently. It often depends on try and error.

What can make a difference is using L1/L2 regularization, since it can shrink features which are „not useful“ to make a good prediction.

I cannot think of a theoretical reason here. I‘m curious if someone comes up with a more theory based answer!

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks! I used L1 regularization as I thought there might be some noisy features in my dataset. I tried Extra Trees too, which has been shown to be robust to noisy feature sets! Still, LR is outperforming all of them (in terms of minority classes). $\endgroup$ – Sarah Aug 30 '19 at 23:04
  • $\begingroup$ This may help better understand the application of class_weight in LogisticRegression: stackoverflow.com/questions/50433130/… $\endgroup$ – Sarah Sep 2 '19 at 21:45
1
$\begingroup$

Seems I found some reasonable justifications for this behavior!

Class_weight in the LogisticRegression approach is applied to sample_weight, which is used in a few internal functions like >_logistic_loss_and_grad, _logistic_loss, etc.:

#Logistic loss is the negative of the log of the logistic function. .
out = -np.sum(sample_weight * log_logistic(yz)) + .5 * alpha * np.dot(w, w)
NOTE: ---> ^^^^^^^^^^^^^

Likewise, in decision-tree based approaches like RandomForest and XGBoosting, the class_weigh is applied to the gini or entropy function, which impacts both nominator and denominator --> less impact on the purity function!

Source code

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.