My question is should you use the same algorithm in feature selection as your model?

If I'm using a KNN model for classification should I also use a KNN algo when running feature selection? Or lightGBM algo for feature selection and lightGBM model?

Or is there a best algo for feature selection that you can use those features with any model?

I know there are several different feature engineering techniques I just don't know which techniques should be used with different machine learning algorithms.

  • 1
    $\begingroup$ there is no theoretical reason not to use same (class of) algorithms both for feature selection and as model $\endgroup$
    – Nikos M.
    Jan 5, 2021 at 18:27
  • 1
    $\begingroup$ In theory, a drawback of using different models could be that features being important to your model being used as a filter to select features might not have the same importance for your model applied during inference $\endgroup$
    – Jonathan
    Jan 5, 2021 at 20:34

1 Answer 1


Yes - You can use the same the same algorithm for both feature selection and prediction. The most common examples are L1 regression and tree-based algorithms. Those algorithms both find the most important features and predict targets using the same fitting mechanism.

The difference between those algorithms and your examples is that those algorithms do both steps concurrently and your examples are sequential.

There is no single best algorithm for feature selection that can also be used with other models for prediction. Different algorithms will perform better on different datasets.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.