9
$\begingroup$

I am working with classification problem on a training data set, which have 100 features. All the features in pairs haven't visible correlation. One can see it in the example pair plot for the some of features:

enter image description here

I am trying to find the right way to decrease the number of features. All the methods of the important feature selection return different sets of features. For example:

  • The MARS returns only five important features, but,
  • Correlation with threshold >1% selects twenty of them.
  • Lasso selects near twenty but selected features are different from the feature set, returned by correlation method.
  • RandomForest selects near twenty-five features different from previous methods.
  • The feature rating in the returned feature sets is different.
  • The PCA method is not applicable, because there are not any visible linear correlated features.
  • etc.

My point is the different tools return different important features. One can combine/union them, one can intersect them. One can use N methods of selection and as a result, one will get N feature sets.

The method: "Check them all" is not applicable there. As an example, one can calculate the time effort if N sets multiplied M prediction models, multiplied 3 rounds of the model tuning. The time effort will be even more if one will union or intersect feature sets. It will take forever! There should be some tactics, an algorithm for the filter a final selection.

How to select the best set of the features if the data set is large, non-correlated and noisy?

Improve this question
$\endgroup$
9
  • $\begingroup$ Have you tried Lasso (L1) Logistic Regression? And how many classes do you have ? $\endgroup$ – Elliot Sep 5 '19 at 8:28
  • $\begingroup$ @Elliot Yes, I tried Lasso with glmnet library. The rating list of the most valuable features by Lasso feature selection is almost the same as result which I achieved by the correlation matrix. $\endgroup$ – Ruben Kazumov Sep 5 '19 at 16:19
  • $\begingroup$ if you're interested in predictive power of your model, you can just search for the best feature selection/model combination. If you're interested in explaining important features, I would go with an union/intersection (if any) of features selected by trees and a lasso. $\endgroup$ – Elliot Sep 6 '19 at 9:42
  • $\begingroup$ @ Elliot I edited the part of the question to clarify it. The time effort of sets combination and the prediction models re-running are huge. $\endgroup$ – Ruben Kazumov Sep 6 '19 at 18:06
  • $\begingroup$ How many observations do you have? Whats the specs of your machine? Did you try packages that use all cores in parallel? $\endgroup$ – Ilker Kurtulus Oct 7 '19 at 17:03
1
$\begingroup$

I believe that what you are looking for is Best Subset, Forward Stepwise, or Lasso?

Here is the R implementation Best Subset

If your dataset is too large, try to subsample your data and then running the feature selection algorithm several times. If you are getting similar results there you go, if the results are different then rank them and select the most frequent ones.

If anyone knows a python implementation for the best subest algorithm, please let me know!

Improve this answer
$\endgroup$
1
$\begingroup$

If the dimensions are not linearly correlated, you may use an autoencoder to perform the dimensionality reduction. Just like PCA that can perform a reconstruction, but with non-linearity. Then, you can perform classification with the latent space.

Autoencoder is a multi-dimensional auto-regressive model with a dimensional bottleneck somewhere in the middle. In general, it contains 2 neural networks, one encoder and one decoder. The encoder takes one data sample as input and it produces an output in latent space with a number of dimensions smaller than the input's. The decoder takes the encoder's output as its input and it tries to produce the original data sample as output. To train an autoencoder, we try to minimize the reconstruction error (the difference between the input sample and the reconstructed sample). The dimensional bottleneck in the middle will force the autoencoder to preserve as much information of the original input as much as possible with a smaller number of dimensions.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Clarify the term autoencoder please. $\endgroup$ – Ruben Kazumov Feb 5 '20 at 3:04
  • $\begingroup$ @RubenKazumov I've edited the answer. $\endgroup$ – koonyook Feb 9 '20 at 3:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.