1
$\begingroup$

I am running a random forest regression with Python's Scikit-Learn, code's below (X - features, y - to be predicted).

# Splitting the dataset into the Training set and Test set
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 1)
    
# Scaling
from sklearn.preprocessing import StandardScaler
sc_X = StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test  = sc_X.transform(X_test)

# Random forest
from sklearn.ensemble import RandomForestRegressor
rf =RandomForestRegressor(max_depth=2, n_estimators = 100, random_state=0)
rf = rf.fit(X_train,y_train)
pred_train = rf.predict(X_train)
pred_test = rf.predict(X_test)

I am running this code for randomly sampled 100k dataset, that has 60+ features. Each time when I check feature importance I get 3 to 4 variables as important (with one of them holding over 80% of importance), and others' importance is set to 0. It is not reasonable to me that only these are important for prediction and the rest is rubbish.

var_num = X_train.shape[1]
plt.barh(range(var_num), rf.feature_importances_, align='center')
plt.yticks(np.arange(var_num), variable_names)
plt.xlabel('Variable Importance')
plt.ylabel('Variable')
plt.show()

Is it possible that I am missing something? That some other parameters needed to be defined? Could this be caused by a high correlation between variables themselves? Or is it really that the rest of my features are useless..?

Improve this question
$\endgroup$
2
$\begingroup$

RandomForestRegressor has a parameter called max_features, which is the number of features to consider when determining the optimal split. You haven't explicitly specified this, so Python will use the default (auto) and consider all features.

Given that your trees are very shallow and you are considering all of the features to split, it would not surprise me that the strongest 3-4 are consistently popping up (the bagging process in random forests will cause some variation within this).

Decreasing max_features and/or increasing max_depth may yield a greater variety of "important" features.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks! I experimented with the max_features and max_depth, as you suggested, and it really increased the number of important features. But it didn't really change my mean absolute error or my variance score. What does this mean for me? Because, I am planning to add new features, and now I am not sure which max_features/max_depth i should "listen to" when it comes to feature importance, in the sense of which variables can I safely exclude. $\endgroup$ – GileBrt Jun 28 '18 at 14:38
  • 1
    $\begingroup$ 1) Random forests implicitly do feature selection, so you don't have to worry about excluding features manually. 2) Very little change in error and variance may suggest that those original 3-4 features are really predictive of the response. 3) The optimal max_features/max_depth depends on what your goals are. If you are only concerned with prediction then feature importance is secondary (and vice versa). If you using the forest for prediction, tune the parameters using a grid search; if for feature selection grow a large forest of shallow trees with conservative (low) max_features. $\endgroup$ – bradS Jun 28 '18 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.