I am using three feature selection method on a dataset containing 15 inputs. I need to extract the best 5 features. Each of the three method gave a subset of the input dataset, but they are different. How do I take the final subset for predicting the target variable?
3
-
2$\begingroup$ Train a model and compare the results for each subset of feature $\endgroup$– LelouchCommented Jul 18, 2023 at 11:54
-
$\begingroup$ Is this a regression or classification problem? $\endgroup$– liakoyrasCommented Jul 18, 2023 at 21:03
-
$\begingroup$ you can train a classic machine learning model and have a look at the feature importances (machinelearningmastery.com/…). This will give you an importance score that will represent how important a given input feature was while for making a prediction. $\endgroup$– LeoCommented May 7 at 11:28
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
You may take one of the following approaches:
Brute Force
You have $ \binom{15}{5} = 3003 $ combinations. If your model training is quick, you may go over all of them and choose the best combination on the score you set.Greedy
You may take a greedy approach.
You find a combination of only 2 features ($ \binom{15}{2} = 105 $) and once you set those 2 you add features one by one.Features Significance Analysis
If you model is linear or if you use trees based model you can get the model rank of the features and use that.
You may use a black box approach by using Shapley Values or other approaches to analysis of the features.Power Predictive Score (PPS Score)
While this is not a global method, it is still useful and for complex models probably much better than correlation based analysis.
$\begingroup$
$\endgroup$
Just to confirm the comment. Use the top features as a method to subset the original data, train the data subset again and then compare the accuracy of each subset for each model performed.
The alternative way in, particularly if you are using regression analysis, is to have independent test samples that have never been through train-test-split. You then apply the different predictive solutions based on the subsets generated by feature selection and perform your own percentage accuracy by comparing the predicted scores against the observed scores.
