I am trying to build a classification model with 1000+ features and I am performing the below steps but I am not sure if it's the correct way to do it

Step 1. Finding Variable Importance using the entire dataset

Step 2. Build a model using the subset of the features returned from Step 1 Once I get my new data I am splitting the data in training and testing and then performing cross validation on training data.

My question is should I use CV while performing variable importance on the entire dataset?

Thanks for your help!


2 Answers 2


Before answering your question I would ask if you actually need so many variables or you can apply some dimensionality reduction technique (i.e. PCA)? It could boost your training speed and maybe result in a better model efficiency.

Lets suppose for some reason (i.e. model explainability) you need the exact variables. You can still remove some of them without using feature importance (i.e. removing highly correlated ones, but there are other alternatives too). Indeed using only feature importance to remove variables can be dangerous. Imagine there is an important factor that highly influence your model but it shows up in several variables. In this case it can happen that the importance of this factor will be distributed among those variables. You will experience maybe that those variables are less important and remove them all thereby also eliminating an otherwise important factor.

To answer your question. I would before step 1 take away my test set and only use the training set for any kind of modeling activity. It's a good practice to guarantee that your test set has never been seen by the model and it did not influence it in any way.

In this case you can just remove less important features and measure the final results on the test set, it's not necessary to do CV.

On the other hand you will probably face with the challenge how to define your threshold for eliminating variables based on feature importance.You can try to do some gridsearch on it. In this case you should avoid measuring the results of different models on the test set or you risk over-fitting on it. In this case using CV can be a good solution to define your threshold.


Cross validation data should only be used to tune parameter values ex. the C term in SVM. Calculate the feature importance on the training set.

  • $\begingroup$ That is incorrect, pre-processing can lead to overfitting and to decide what features should be used needs to be done "blindly", thus it is a good idea to use CV. $\endgroup$
    – JahKnows
    Commented Nov 3, 2018 at 13:52
  • $\begingroup$ Sorry if I'm wrong, but isn't overfitting a problem with any ML algorithm that runs on the training set? If you are running something like logistic regression, you basically assign some form of weighting or 'importance' to your features. This would be run on the training set. $\endgroup$
    – Anshul G.
    Commented Nov 3, 2018 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.