0
$\begingroup$

I ran Boruta feature selection prior to XGB training\testing step and didn't see any difference, although ~30/200 features were rejected prior to going into the training. Can it be that internal feature selection of XGB is comparable to Boruta step and assigns nearly 0 importance to the same 30 features?

My logic is simple - because these 200 features are the initial dataset it is very hard to believe it is also the optimal set. Therefore feature selection should bring some extra performance, except it is not.

How do I test the above assumption? If I take a less smart model like logistic regression and test it with and without Boruta - can I expect significant difference in f1 or auc?

Surely, someone already answered this question before systematically... any good links to papers? If I have to say to my team that we don't need feature selection step, I can use all support :)

$\endgroup$

1 Answer 1

2
$\begingroup$

Many aspects are at play here, mostly related to the data itself.

Because with most feature selections, they are usually there to perform redundancy reduction. The redundancy is mostly defined with respect to the correlation or relatedness of your features.

So, for example, if there are two features that are very highly correlated to each other, then most likely, the split finding within XGB will find the same things, so nothing is gained from using either of those features. This is due to the nature of the algorithm itself. So then the result of the model before and after feature selection will remain the same.

However, it will lead to some computational speed up as through feature selection you've reduced the quantity of computations that need to be performed.

However, for linear algebra-based models, you could find that reducing the features could result in bigger changes compared to the tree models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.