I was wondering does it make sense to use random forest to select most important variables then put into logistic regression for prediction? I think that it might not make sense because what's important for random forest might not be important for logistic regression?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
There are many factors which are underlying the 'importaces' of featues obtained from random forest.
For instance, features with more number of categories (unique values if its a numerical feature) would be more likely to get find splits; making it more important feature. Having all features with same number of categories is not a common scenario.
So, even though the features identified are likey to be the best predictors (in logistic regressions etc.), one need to take the inherent biases in the randome forest algorithm into consideration while utilizing the 'important' features. Therefore whether it would make sense to use the important features from random forest model into logistic regression would depend on if any and how much the importances are biased.
Source for the information and further useful details on the biases in estimation of feature importances by random forest algorithm: Bias in random forest variable importance measures: Illustrations, sources and a solution Bias in random forest variable importance measures: Illustrations, sources and a solution
$\begingroup$
$\endgroup$
There is not any single good answer to this question as this thing depends on many factors and one of them is the data. As per my experience, I prefer not to remove features as even little information can be very useful.
$\begingroup$
$\endgroup$
The Boruta algorithm uses random forests to select relevant features. There are Python Github and R package implementations.
