1
$\begingroup$

I have about 30 variables and trying to create a Random Forest model. All the variables are expected to be predictors of outcome. I want to find the best model based on a C-stat score with any number of features. Shall I do feature selection first or hyperparameter tuning? I read that people prefer feature selection first, but they do have a few hundred features. I only have 30.

I am leaning towards hyperparameter tuning before going into feature selection.

$\endgroup$

1 Answer 1

0
$\begingroup$

Feature selection and hyper-parameter tuning have different purposes.

If you consider the entire workflow of data to the ML model and what the model is doing you can get to some intuition what to do first. The model will learn some patterns that are in your data, and the hyper-parameter will slightly dictate what the patterns that will be learned are, especially in a random forest. So, the hyper-parameter tuning will depend on the features that you have.

The act of feature selection will be before the model training, the feature selection results will have a direct change on the hyper-parameters of the model itself. So, in theory you should do feature selection before hyper-parameter tuning as there may be patterns that you do not want your model to learn which can be removed. Or redundencies within the data can be removed so the hyper-parameter tuning could be faster.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.