Say I want to do features selection on a sparse matrix, i.e., 10,000 rows x 1500 features, but the matrix is mostly sparse. Let's say the features are all numeric and the target is binary and discrete.

What's the correct and efficient way to apply feature selection? Moreover, I'm interested in applying mutual information on it.


2 Answers 2


You can do a dimentionality Reduction as your matrix is Sparse. I would suggest to use PCA. PCA will reduce your 1500 input into k dimensional input of your choice with as much information retained as possible . Here k is a hyperparameter that you need to tune and fine the best one.

Another Approach is LASSO classfier which is a linear model with L1 regularization. This model will perform automatic feature selection and zero out the weight of feature that is not needed. But your input columns must be independent.

  • $\begingroup$ Is there a reason to choose PCA over something like calculating the correlations between the features and the target and then picking the top N features with the highest correlation? $\endgroup$ Commented Jan 11, 2021 at 6:08
  • $\begingroup$ PCA will reduce your dimension along with keeping as much linear relations as possible. And For Training ML models your feature ideally be independent. Most of the ML models perform poorly if there is some correlation among inputs. $\endgroup$
    – SrJ
    Commented Jan 11, 2021 at 7:53
  • $\begingroup$ @SrJ Does not answer why PCA is preferred when the data is sparse. $\endgroup$
    – lpounng
    Commented Nov 21, 2022 at 10:17

Perhaps someone else can contribute since I'm an amateur, but here's what I would do.

It sounds like you are interested in classification, so let's consider that. In general I don't think its guaranteed that feature selection will improve your model (with respect to some performance measure, e.g., accuracy). So we want to determine if feature selection can improve our baseline model of using all features. This could be the case if some features are redundant, noisy, irrelevant, etc.

There are a lot of ways to do feature selection, and it isn't known a priori which will be best, so you should try a few (and it should be part of a larger pipeline). For mutual information (MI), you calculate the MI between each feature and the target variable. Then you choose the top k features based on their MI -- you can treat k like a hyperparameter and optimize for it (along with other hyperparameters) using nested CV.

The result of nested CV will be a sample distribution of the generalization error estimate. You can compare this distribution with the baseline models using students t-test to determine if there is a statistically significant improvement

Now suppose the dataset were complex and that it wasn't possible to train a model using all of the features. In this case we can't take as our baseline the model using all features. Instead, we can compare different feature selection pipelines. For example, we start with using MI, find the best hyperparameters and estimate the generalization error using nested CV (which will give a distribution as before). Now we consider a different technique, e.g., recursive feature elimination and estimate the generalization error using nested CV. Finally, we compare these two distributions using the students t-test as before.

We aren't limited to comparing two different techniques either (e.g. RFE vs MI) but we can also compare different pipelines (e.g. RFE vs PCA + MI or t-SNE + RFE vs MI,etc.)

Hope this helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.