I've recently started playing around with Datasets on Kaggle and am struggling to bring structure into the feature selection part of my predictive model building process.

I end up trying most or all of the following (with varying cutoff points depending on the method) and then running my model of choice on each different selected subset of features to see what works best.

  • forward/backward/mixed selection
  • eliminate high VIF features
  • eliminate low mutual information features
  • PCA
  • Partial Least Squares
  • Lasso regression

This is time consuming and I'm wondering if there are any rules of thumb or tips on which methods work best in which situations? For example is likely that for large numbers of features high VIF elimination leads to deleting too many vars, and PCA is usually good? Is Lasso always better than VIF? Any tips would be appreciated!



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.