Can you please explain when (under what circumstances) the dimensional reduction and/or feature selection techniques are not effective to mitigate the curse of dimensionality?

  • 1
    $\begingroup$ The data might not be embedded in a lower dimensional manifold; it might already be as compressed as possible. (Why can't you effectively compress a ZIP file?) $\endgroup$
    – Emre
    Nov 29, 2017 at 19:06

2 Answers 2


This is helpful when the data is not understandable and you don't have any data dictionary or you have too many columns which doesn't make any sense even after through investigation, then it is wise to go for Dimensionality Reduction.

There is very good chance that you loose information at granular level, for example you gave 100 columns and you got 10 PC's(Applied PCA) which can explains most of the Data. As you can only get that much out of that technique.

Feature Selection: If you miss even a single feature which is significant WRT target variable but you overlooked that feature then your model might not be able explain the most, you are even satisfied with the result but still there is window of improvement. You should be very careful and you need check as many times(many iterations) as possible to build a good/decent model and get the best results out of it.

I hope this may help you.

  • $\begingroup$ Repeating your approach over many iterations, there is a high probability you'd find something that fits the data. And that's called data snooping. $\endgroup$
    – horaceT
    Mar 2, 2018 at 5:55
  • $\begingroup$ @canberk_gurel: was my answer helpful? $\endgroup$
    – Toros91
    Mar 2, 2018 at 6:35
  • $\begingroup$ Beware this is a dangerous approach. Whatever features you find are likely to have little predictive power. $\endgroup$
    – horaceT
    Mar 2, 2018 at 6:39

Both approaches are 'lossy'.

For PCA, assuming you retain fewer components than variables, you necessarily throw away information. If you don't have a few components that capture most of the variance, you could throw a lot away.

For feature selection, (I assume you mean an automated approach like lasso), you again throw away information, hopefully only incremental so. But, depending on how you structure the selection routine, you let an algorithm make some level of design decisions which result in less data.

Whether this is bad or not depends on intent. Fire pure classification it's less an issue. For interpretability it can be essential.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.