1
$\begingroup$

I am doing feature selection using Sklearn:

  • Tree-based feature selection : RandomForestClassifier.feature_importances_
  • L2-based feature selection: LogisticRegression.coef_

Target variable is binary classes. The training set is standardized.

How should I interpret when a certain feature shows significant importance in Random Forest estimator, but negative coefficient in Logistic Regression?

$\endgroup$
  • 1
    $\begingroup$ A large negative coefficient in LR (assuming you've scaled the data) implies that the feature is important, and that a larger value of that feature corresponds to smaller probability of being in the positive class. "Different results" would be more like near-zero coefficient in LR and high importance in RF (and vice versa) $\endgroup$ – Ben Reiniger Sep 19 at 13:41
3
$\begingroup$

Negative coefficient in logistic regression means negative relationship between predictor variable and the response variable.

Example- in a model price can be a predictor and will have negative relationship with binary response variable product purchased or not.

And neagtive coefficient in logistic regression does not mean relationship has low strength, it only means changes in predictor has a reverse effect on response variable and if coefficient is highly negative it means feature is very important and small changes in it impacts response but in reverse direction.

Feature importance does not tell you nature or direction of relationship but only tells you strength of relationship so they are never negative. Coefficients in logistic regression tells you both strength and direction or nature (positive or negative) of relationship. Also high importance in random forest means strong relationship between predictor and response but importance column derived from tree based models is silent on nature or direction.

Hope this helps.

$\endgroup$
  • $\begingroup$ That makes sense to me now. I truly appreciate your taking the time to write the answer. :) $\endgroup$ – Hani Sep 19 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.