1
$\begingroup$

The importance metric in random forests is a way to determine the significance of a predictor variable in a model. It does this by randomly permutating the values of one predictor variable at a time and observing how much it affects the model's performance.

Instead, why don't we remove the variable, retrain the model and measure the effect?

I am guessing it may be because removing the variable from the model reduces its "complexity" (measured by the number of predictors) and this may have an effect on its performance. But I am not sure if this is the reason, or if there are other reasons besides this one.

$\endgroup$

1 Answer 1

0
$\begingroup$

The importance metric in random forests is based on the idea of permuting the values of one predictor variable at a time to assess its significance in the model. This approach has several advantages over removing the variable and retraining the model to measure the effect:

  1. Consistency with Model Structure: Random forests are ensemble models composed of multiple decision trees. The structure and architecture of the model remain unchanged when using permutation-based importance. Removing a variable, on the other hand, changes the model's structure. Permutation-based importance allows you to evaluate the variable's importance within the context of the existing model.

  2. Model Complexity and Interactions: Removing a variable not only affects the model's complexity (number of predictors) but can also disrupt interactions and relationships between variables. Permutation-based importance isolates the variable's contribution, while variable removal could lead to a loss of information and interactions with other predictors.

  3. Non-Destructive: Permutation-based importance does not alter the original dataset. Removing a variable from the dataset can be problematic if the dataset is needed for other analyses or if you later realize the variable was important and want to include it in the analysis.

  4. Consistency with Other Importance Metrics: Permutation-based importance is consistent with other importance metrics like Gini impurity or mean decrease in accuracy, which are commonly used in decision trees and random forests. This consistency allows you to compare variable importance across different models and datasets.

$\endgroup$
3
  • $\begingroup$ Could you elaborate on (or point to resources that support) point #4 related to how permutation importance is consistent with other importance metrics? Its not clear how this might be the case. $\endgroup$
    – jluchman
    Jan 22 at 21:22
  • $\begingroup$ I believe a fifth would also be that a permutation approach does not require re-fitting the model and the time it would take to do so. The time required to re-fit the model while omitting each feature (i.e., one refit model per feature) can be substantial and can make an omitted feature approach impractical. $\endgroup$
    – jluchman
    Jan 22 at 22:06
  • $\begingroup$ Consistency, in this context, means that the rankings or assessments of variable importance obtained through permutation-based methods usually align with those obtained through other established metrics. $\endgroup$
    – technik
    Jan 23 at 9:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.