I'm currently using Random Forest to train some models and interpret the obtained results.

One of the features I want to analyze further, is variable importance. The thing is I am not familiar on how to do a proper analysis of the results I got. Let's say I have this table:

| Predictor  | Importance |
|   var_1    |   num_1    |
|   var_n    |   num_n    |

What is a proper analysis that can be conducted on the values obtained from the table, in addition to saying which variable is more important than another?

I was suggested something like variable ranking or using cumulative density function, but I am not sure how to begin with that.

  • 1
    $\begingroup$ There are multiple ways of calculating variable importance, some more reliable than others. Without any other information provided, you should be wary of trying to glean anything aside from a vague ranking of the features. What does the documentation say about how the importance is calculated? $\endgroup$ – dsaxton Jul 10 '18 at 22:35
  • $\begingroup$ @dsaxton what I'm trying to understand is what kind of analysis can I conduct from a feature importance table besides saying which one is more important. $\endgroup$ – nicodp Jul 10 '18 at 22:50

I would be reluctant to do too much analysis on the table alone as variable importances can be misleading, but there is something you can do. The idea is to learn the statistical properties of the feature importances through simulation, and then determine how "significant" the observed importances are for each feature. That is, could a large importance for a feature have arisen purely by chance, or is that feature legitimately predictive?

To do this you take the target of your algorithm $y$ and shuffle its values, so that there is no way to do genuine prediction and all of your features are effectively noise. Then fit your chosen model $m$ times, observe the importances of your features for every iteration, and record the "null distribution" for each. This is the distribution of the feature's importance when that feature has no predictive power.

Having obtained these distributions you can compare the importances that you actually observed without shuffling $y$ and start to make meaningful statements about which features are genuinely predictive and which are not. That is, did the importance for a given feature fall into a large quantile (say the 99th percentile) of its null distribution? In that case you can conclude that it contains genuine information about $y$. If on the other hand the importance was somewhere in the middle of the distribution, then you can start to assume that the feature is not useful and perhaps start to do feature selection on these grounds.

Here is a simulation you can do in Python to try this idea out. First we generate data under a linear regression model where only 3 of the 50 features are predictive, and then fit a random forest model to the data. Now that we have our feature importances we fit 100 more models on permutations of $y$ and record the results. Then all we have to do is compare the actual importances we saw to their null distributions using the helper function dist_func, which calculates what proportion of the null importances are less than the observed. These numbers are essentially $p$-values in the classical statistical sense (only inverted so higher means better) and are much easier to interpret than the importance metrics reported by RandomForestRegressor. Or, you can simply plot the null distributions and see where the actual importance values fall. In this case it becomes very obvious that only the first three features matter where it may not have been by looking at the raw importances themselves.

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor

# number of samples
n = 100
# number of features
p = 50
# monte carlo sample size
m = 100

# simulate data under a linear regression model
# the first three coefficients are one and the rest zero
beta = np.ones(p)
beta[3:] = 0
X = pd.DataFrame(np.random.normal(size=(n, p)), 
                 columns=["x" + str(i+1) for i in range(p)])
y = np.dot(X, beta) + np.random.randn(n)

# fit a random forest regression to the data
reg = RandomForestRegressor()
reg.fit(X, y)

# get the importances
var_imp = (pd.DataFrame({"feature": X.columns, 
                        "beta": beta, 
                       "importance": reg.feature_importances_}).
           sort_values(by="importance", ascending=False).

# fit many regressions on shuffled versions of y
sim_imp = pd.DataFrame({c: np.empty(m) for c in X.columns})

for i in range(m):
    reg.fit(X, np.random.permutation(y))
    sim_imp.iloc[i] = reg.feature_importances_

# null distribution function
def dist_func(var, x):
    return np.mean(sim_imp[var] < x)
  • $\begingroup$ If I get you correctly, then you are trying to say to shuffle the cols values randomly and iterate the model given no of times and them calculate the real feat imp, right? Won't we do this generally for Tree based models? Thanks for a wonderful answer(+1) $\endgroup$ – Aditya Jul 11 '18 at 1:27
  • $\begingroup$ What I understood is shufling the y row so the labels do not correspond to the real values of each variables' row, but the cols values remain intact (just with wrong labels). Then fit the model n times with this shuffled train data. Also (+1) $\endgroup$ – nicodp Jul 11 '18 at 1:35
  • $\begingroup$ @dsaxton thanks for this detailed answer! I haven't understand very well the last paragraph though. Could you elaborate it with an example if it's not too much to ask? Thank you anyway! $\endgroup$ – nicodp Jul 11 '18 at 1:36
  • 1
    $\begingroup$ @Aditya What's often done to calculate importance for tree-based models is to shuffle the $x$'s, but here we are actually shuffling $y$, which means none of the features are important. This helps to interpret the importances that we actually observe. $\endgroup$ – dsaxton Jul 11 '18 at 3:11
  • 1
    $\begingroup$ @nicodp I added a bit more with a simulation, let me know if that helps to clarity. $\endgroup$ – dsaxton Jul 11 '18 at 3:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.