I am working with classification problem on a training data set, which have 100 features. All the features in pairs haven't visible correlation. One can see it in the example pair plot for the some of features:
I am trying to find the right way to decrease the number of features. All the methods of the important feature selection return different sets of features. For example:
- The MARS returns only five important features, but,
- Correlation with threshold >1% selects twenty of them.
- Lasso selects near twenty but selected features are different from the feature set, returned by correlation method.
- RandomForest selects near twenty-five features different from previous methods.
- The feature rating in the returned feature sets is different.
- The PCA method is not applicable, because there are not any visible linear correlated features.
- etc.
My point is the different tools return different important features. One can combine/union them, one can intersect them. One can use N methods of selection and as a result, one will get N feature sets.
The method: "Check them all" is not applicable there. As an example, one can calculate the time effort if N sets multiplied M prediction models, multiplied 3 rounds of the model tuning. The time effort will be even more if one will union or intersect feature sets. It will take forever! There should be some tactics, an algorithm for the filter a final selection.
How to select the best set of the features if the data set is large, non-correlated and noisy?
glmnetlibrary. The rating list of the most valuable features by Lasso feature selection is almost the same as result which I achieved by the correlation matrix. $\endgroup$ – Ruben Kazumov Sep 5 at 16:19