I have a list of variables from which I would like to train a Random Forest Algorithm. I suspect that a few of my input variables, which have noisy distributions, won't be able to predict much. Can I use them anyway, knowing the algorithm will eliminate them in the process, or should I beware that these variables may biais my model?


Random Forest tends to be not too sensitive to features with low predictive power. The reason is that RF looks for a "best split" given a subset of features (columns) and observations (rows) at each node. So "weak" features will likely be ignored in most cases (splits).

However, removing the $x$ percent weakest features may increase the model's performance. In case you use sklearn, there are convenience functions to do this, e.g. SelectFromModel(). See the docs for more details.

>>> from sklearn.ensemble import ExtraTreesClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.feature_selection import SelectFromModel
>>> X, y = load_iris(return_X_y=True)
>>> X.shape
(150, 4)
>>> clf = ExtraTreesClassifier(n_estimators=50)
>>> clf = clf.fit(X, y)
>>> clf.feature_importances_  
array([ 0.04...,  0.05...,  0.4...,  0.4...])
>>> model = SelectFromModel(clf, prefit=True)
>>> X_new = model.transform(X)
>>> X_new.shape               
(150, 2)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.