I am currently using feature selection approaches like filter, wrapper, embedded etc.

All these methods give different set of features and I rank them based on their frequency of occurrence in other feature selection approach.

Ex: If Age occurs in all 6 feature selection algorithm, then they will have a rank as 6. If gender occurs only in 2 feature selection algorithm, its rank would be 2.

So I arrange them in descending order and choose the features which has occurred in at least 3-4 feature selection algorithm.

But where I am trying to seek your help is

1) Is there any systematic way where I get only a subset of features which returns best output? I thought genetic algorithm for feature selection will return a feature subset which will give high output. But unfortunately it lists all features with their importance. I don't wish to define a threshold myself to select few. Example is currently Xgboost produces an F1-score of 81 with around 27-28 features but when I manually play around, it gives around 81.9 or 82.1 with just 6-7 features. So I would like to have a way to select this 6-7 features automatically/systematically without human intervention

2) Is there any algorithm, like RFE but which does exhaustive search like genetic algorithm and finally returns us the best feature set? Mix of RFE and genetic algorithm is what I need

3) I wish to have an exhaustive search in feature space and finally provide me the best feature set which will provide best f1-SCORE which is my objective.

I am looking for a systematic way/algorithmic way to do this rather than me deciding to pick features which occurs at least in 3 feat selection algorithms

Hope my question is clear. Can you help me with this?


1 Answer 1


I believe that what you are looking for is Best Subset, Forward Stepwise, or Lasso?

Here is the R implementation Best Subset

If anyone knows a python implementation, please let me know!

  • $\begingroup$ Hi, thanks for the response. upvoted.Does Best subset applies to Xgboost? $\endgroup$
    – The Great
    Jan 4, 2020 at 6:27
  • $\begingroup$ I understand that yes, but It is extremely expensive (check page 11) $\endgroup$ Jan 4, 2020 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.