Which feature selection technique to pickup(Boruta vs RFE vs step selection)

Question

I have data with 103 columns. I would like to understand which algorithm is best for feature selection and what may be the logic to call any feature as best.

I run below feature selection algorithms and below is the output:

    1) Boruta(given 11 variables as important)
    2) RFE(given 7 variables as important)
    3) Backward Step Selection(5 variables)
    4) Both Step Selection(5 variables)

I not able to decide which one to pick up; with domain knowledge it appears I must pick up results from Boruta (as it is giving most number of variables and all seems important).

However I don't find any concrete reason to pickup the best combination.

Answer 1

There is a tradeoff between selecting features and precision. Fewer features probably have less precision (predictive power).

Select the features that make sense for your problem taking into consideration the trade-off of information vs performance. The fewer features the model sees the predictive power that it has.

Answer 2

Some algorithms perform feature selection inherently - e.g. LASSO, random forests, and gradient-boosted models like XGBoost and LightGBM. If you are using those then there is no need for manual feature selection.

However if you go down the feature selection route, it maybe good to start with features which have been suggested by all the approaches you have tried (if there are any).

Choosing the "best" feature really depends on your goal and setting.

• If you are chasing modelling accuracy, then the "best" feature might be the one which improves model accuracy the most when included in the model.
• If you are trying to reduce the data dimension, the "best" feature might be the one which captures the most information (e.g. a timestamp will capture year, month, day, hour).
• If you are going to dynamically retrieve features in live production, the best feature might be the one which adds the most accuracy to the model given its availability and cost / time.