Features reduction for the not correlated data set

Question

I am working with classification problem on a training data set, which have 100 features. All the features in pairs haven't visible correlation. One can see it in the example pair plot for the some of features:

I am trying to find the right way to decrease the number of features. All the methods of the important feature selection return different sets of features. For example:

• The MARS returns only five important features, but,
• Correlation with threshold >1% selects twenty of them.
• Lasso selects near twenty but selected features are different from the feature set, returned by correlation method.
• RandomForest selects near twenty-five features different from previous methods.
• The feature rating in the returned feature sets is different.
• The PCA method is not applicable, because there are not any visible linear correlated features.
• etc.

My point is the different tools return different important features. One can combine/union them, one can intersect them. One can use N methods of selection and as a result, one will get N feature sets.

The method: "Check them all" is not applicable there. As an example, one can calculate the time effort if N sets multiplied M prediction models, multiplied 3 rounds of the model tuning. The time effort will be even more if one will union or intersect feature sets. It will take forever! There should be some tactics, an algorithm for the filter a final selection.

How to select the best set of the features if the data set is large, non-correlated and noisy?

Answer 1

I believe that what you are looking for is Best Subset, Forward Stepwise, or Lasso?

Here is the R implementation Best Subset

If your dataset is too large, try to subsample your data and then running the feature selection algorithm several times. If you are getting similar results there you go, if the results are different then rank them and select the most frequent ones.

If anyone knows a python implementation for the best subest algorithm, please let me know!

Answer 2

If the dimensions are not linearly correlated, you may use an autoencoder to perform the dimensionality reduction. Just like PCA that can perform a reconstruction, but with non-linearity. Then, you can perform classification with the latent space.

Autoencoder is a multi-dimensional auto-regressive model with a dimensional bottleneck somewhere in the middle. In general, it contains 2 neural networks, one encoder and one decoder. The encoder takes one data sample as input and it produces an output in latent space with a number of dimensions smaller than the input's. The decoder takes the encoder's output as its input and it tries to produce the original data sample as output. To train an autoencoder, we try to minimize the reconstruction error (the difference between the input sample and the reconstructed sample). The dimensional bottleneck in the middle will force the autoencoder to preserve as much information of the original input as much as possible with a smaller number of dimensions.