I am interested in learning what routine others use (if any) for Feature Reduction/Selection.

For example, If my data has several thousand features, I typically try {2,3,4} things right away depending on circumstances.

  1. Zero variance/Near zero variance

    • Using R package caret, nzv
    • I find a v.small percentage is zero variance and a few more are near zero variance.
    • Then by using nzv$PercentUnique I may remove the bottom quartile of features depending on the range of PercentUnique's.
  2. Correlation to find multicollinearity

    • I find the correlation matrix and remove values > 0.75 and remove.
    • I have seen others use correlations > 0.5 or 0.6, but don't have any references for it.
  3. Boruta / Random Forest

    • Love Boruta package but it takes a while.
    • Then here again use Forward Feature Selection.
  4. PCA

    • Depending on the nature of the data I will try PCA last.
    • If the model must be explainable then I skip this.
    • I may use several criteria: 80, 90, 95% error explained
    • Forward Feature selection, look for first ~3:10 orthogonal features

NOTE: I am not suggesting this is the best/worst routine but I'm opening the floor to civil debate. If you need a definition for Civil Debate see Wikipedia.

  • 1
    $\begingroup$ honest question, but stackexchange sites like datasciense.se are not for debating things but for objectively defined questions and answers $\endgroup$
    – Nikos M.
    Aug 22, 2020 at 6:30
  • $\begingroup$ @nikos-m After thinking about your post, I remembered this, datascience.stackexchange.com/questions/34357/… Granted, It is not a code related question like S.E.but on topic. I'm torn... $\endgroup$
    – mccurcio
    Aug 22, 2020 at 17:58
  • $\begingroup$ This older post discusses the merits of procedures, datascience.stackexchange.com/questions/14864/… The question becomes, 'are people willing to broaden the scope here?' $\endgroup$
    – mccurcio
    Aug 22, 2020 at 18:13


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.