I was reading An Introduction to Statistical Learning when I came across the High Dimensional section, they argue that high-dimensional data causes a lot of problems. My question is, it is necesary to have more features than observations to apply any feature selection model?

If so, can you provide me with any article that talks about it?

  • $\begingroup$ Where did you see that you need more features that observations ? $\endgroup$
    – lcrmorin
    Nov 25, 2020 at 19:06

1 Answer 1


No matter if the number of features is bigger or smaller of the number of samples, applying feature selection to identify the features with the biggest influence before proceeding with the modeling improves the model performance and helps prevent overfitting and eliminate noise bias (of course, in cases where the sample size is smaller than the number of features the need for dimensionality reduction is even bigger!).

Keep in mind that feature selection is not the only way to achieve dimensionality reduction, it can also be achieved with the extraction of new, more compact features from the initial ones (the only negative thing with that is that you cannot have clear physical interpretation of the selected features).

It is not clear from you question if you have in mind some regression or classification task (or just generally asking), but in both cases the general logic is the same. Nevertheless, I have provided examples for both:

  • Here is an example applying most common techniques for dimensionality reduction for classification (including feature selection ones) on the Iris dataset.

  • Here is an example applying feature selection for linear regression on a sample of size 1000 and 100 features.

Also, here is some additional reading in case you want to undertand better the whole dimensionality reduction subject:

  • An article discussing dimensionality reduction and suggesting alternative methods.

  • In this link, dimensionality reduction (pluses and minuses) is again discussed and the most popular linear (feature extraction, no feature selection!) method - PCA - is explained.

  • A review paper discussing state of the art methods for Feature Selection for Classification can be found here.

I hope this helped! :)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.