I am working on a binary classification with 977 rows using different algorithms

I am planning to select important features using wrapper methods.

As you might know, wrapper methods involve use of ML model to find the best subset of features.

Therefore, my question is as below

a) Should I use best hyperparameters even for feature selection using ML model? If yes, why?

b) If no to above question, then am I right to understand that we use best hyperparameters using model building with important features selected above (using a wrapper method)?

c) Is it normal to expect that best hyperparameters from a) and b) should be the same? because both are trying to solve the same objective (best f1-score for example)

Am not sure but feel that doing grid search to find best hyperparameters for feature selection and model building separately seems to be a overkill.

can share your views on this please?


1 Answer 1


Feature selection can have an impact of hyperparameters which are optimal and vice versa. So if you want to squeeze every bit of performance from your model you should do both of the together. But as you rightly mentioned that it may not be feasible and i would suggest the following step :

Very loosely optimize hyperparameters, just to make sure you don't assign extremely bad values to some hyperparameters. This can often just be done by hand if you have a good intuitive understanding of your hyperparameters, or done with a very brief hyperparameter optimization procedure using just a bunch of features that you know to be decently good otherwise.

Feature selection, with hyperparameters that are maybe not 100% optimized but at least not extremely terrible either. If you have at least a somewhat decently configured machine learning algorithm already, having good features will be significantly more important for your performance than micro-optimizing hyperparameters. Extreme examples: If you have no features, you can't predict anything. If you have a cheating feature that contains the class label, you can perfectly classify everything.

Optimize hyperparameters with the features selected in the step above. This should be a good feature set now, where it actually may be worth optimizing hyperparams a bit.

Please refer to this link for more details


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.