# Random forest implementation with probability of choosing column or guarantee of choosing set of columns

Are there random forest implementations which allows for choosing set of columns which will be always selected for every tree in the forest ? or implementations allowing for specifying probabilities of choosing each column ?

Both cases could be simulated by choosing variables (partially or fully randomly) and building tree containing all variables in new temporal dataset, then repeating this procedure until we obtain assumed number of trees and then merging trees into forest,

but there are drawbacks like moving large number of data in scripting language until they are sent to low level implementation of forest for building tree or lack of merging procedures, which makes work with set of trees harder.

Ranger in R can do the first request. The always.split.variables argument defines which columns should always be included.

• ranger also has the split.select.weights argument which allows you to set the probabilities of selecting each column. – bradS Nov 14 '18 at 21:54
• It does! Why didn't I see that. – timcdlucas Nov 15 '18 at 8:32

I know that neither xgboost, nor sklearn offer what you want. I have checked R, and didn't find it either.

But random forests are easy models to implement, so you can just produce one by yourself:

Here, in python using sklearn:

import numpy as np
from sklearn.tree import DecisionTreeClassifier
from scipy.stats import mode

class MyRandomForest:
def __init__(self, Pcol, Pobs, n_estimators=10):
self.n_estimators = n_estimators
self.Pcol = Pcol  # vector
self.Pobs = Pobs  # scalar

def fit(self, X, y):
self.classes_ = np.unique(y)
assert len(self.Pcol) == X.shape[1]
self.cols = []
self.ms = []
while True:
j = np.random.rand(X.shape[1]) <= self.Pcol
if not np.any(j):  # at least one column must be chosen!
continue
x = X[:, j]
i = np.random.choice(range(len(X)), int(len(X)*self.Pobs), False)
self.cols.append(j)
self.ms.append(DecisionTreeClassifier().fit(x[i], y[i]))
if len(self.ms) == self.n_estimators:
break
return self

def predict(self, X):
yp = [m.predict(X[:, cs]) for cs, m in zip(self.cols, self.ms)]
yp = mode(yp, 0)[0][-1]
return yp

if __name__ == '__main__':  # TEST
from sklearn.cross_validation import StratifiedKFold
from sklearn.metrics import accuracy_score
X = iris.data
y = iris.target
for tr, ts in StratifiedKFold(y):
m = MyRandomForest([0.5, 1, 0.2, 0.3], 1).fit(X[tr], y[tr])
print(accuracy_score(y[ts], m.predict(X[ts])))


If you are using Linux, you can use multiprocessing to easily make it run in parallel.

What I did was just:

1. train a sequence of individual trees, using your probabilities
2. save both the models and whichever columns were sampled for training because we want to use only those for evaluation (or we will get an error)
3. using mode to get the most voted prediction

Note: Pobs is the fraction of observations to use. You may also want to change np.random.choice(..., ..., False) to np.random.choice(..., ..., True), or make it configurable, to allow for bootstrap with resampling. Usually, random forests are trained using Pobs=1 and resample=True.