Variable Importance Random Forest on R

Question

I am currently using a random forest model for classification, however I am unsure how the feature selection technique "varImp" works on R. I understand the context of variable importance, however when I implement it in R it doesn't seem to produce the results I expect.

When removing the most important variable (of 31 features), the model's accuracy does not decrease. I would expect it to as it should be contributing the most to the model's ability to classify.

Could someone please explain what this function is doing?

Answer 1

What the function is doing for each variable?

1. Record the Out-Of-Bag (OOB) accuracy for each tree.
2. "shuffle" or permute the values of that variable. This means you take all the values of that variable in the data, and assign those values randomly back out to the observations, which is a way of introducing noise and getting rid of the signal that that variable provided.
3. Now it finds the OOB accuracy again, but this time the values for that variable are incorrect since we permuted it. By introducing noise where your model expects signal, you should see a decrease in performance.
4. Compare the original accuracy in (1) to the accuracy in (3) for each variable. If the model performance decreases a lot for a variable in step (3) compared to (1), then it is deemed to have greater importance.

Why does removing the most important variable not have a negative affect on accuracy? (my guess)

Probably because that important variable is correlated with some other variable(s) you have. Your model can capture the information contained in that missing important variable by using a few other variables to make up for it. When you drop the important variable, which other variables see a notable gain?

Answer 2

The VarImp works on the principle how your variables are helping data to split with minimum error and order them by their efficiency.

As per you said even after removing the important variable you are still getting same accuracy .May be it is not affecting your dependent variable to large extent or you didn't perform needed transformations.

There are other ways to find feature importance like recursive feature elimination.Here is description RFE in caret