5
$\begingroup$

I am working with classification problem on a training data set, which have 100 features. All the features in pairs haven't visible correlation. One can see it in the example pair plot for the some of features:

enter image description here

I am trying to find the right way to decrease the number of features. All the methods of the important feature selection return different sets of features. For example:

  • The MARS returns only five important features, but,
  • Correlation with threshold >1% selects twenty of them.
  • Lasso selects near twenty but selected features are different from the feature set, returned by correlation method.
  • RandomForest selects near twenty-five features different from previous methods.
  • The feature rating in the returned feature sets is different.
  • The PCA method is not applicable, because there are not any visible linear correlated features.
  • etc.

My point is the different tools return different important features. One can combine/union them, one can intersect them. One can use N methods of selection and as a result, one will get N feature sets.

The method: "Check them all" is not applicable there. As an example, one can calculate the time effort if N sets multiplied M prediction models, multiplied 3 rounds of the model tuning. The time effort will be even more if one will union or intersect feature sets. It will take forever! There should be some tactics, an algorithm for the filter a final selection.

How to select the best set of the features if the data set is large, non-correlated and noisy?

$\endgroup$
  • $\begingroup$ Have you tried Lasso (L1) Logistic Regression? And how many classes do you have ? $\endgroup$ – Elliot Sep 5 at 8:28
  • $\begingroup$ @Elliot Yes, I tried Lasso with glmnet library. The rating list of the most valuable features by Lasso feature selection is almost the same as result which I achieved by the correlation matrix. $\endgroup$ – Ruben Kazumov Sep 5 at 16:19
  • $\begingroup$ if you're interested in predictive power of your model, you can just search for the best feature selection/model combination. If you're interested in explaining important features, I would go with an union/intersection (if any) of features selected by trees and a lasso. $\endgroup$ – Elliot Sep 6 at 9:42
  • $\begingroup$ @ Elliot I edited the part of the question to clarify it. The time effort of sets combination and the prediction models re-running are huge. $\endgroup$ – Ruben Kazumov Sep 6 at 18:06
  • $\begingroup$ How many observations do you have? Whats the specs of your machine? Did you try packages that use all cores in parallel? $\endgroup$ – Ilker Kurtulus Oct 7 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.