Random forest implementation with probability of choosing column or guarantee of choosing set of columns

Question

Are there random forest implementations which allows for choosing set of columns which will be always selected for every tree in the forest ? or implementations allowing for specifying probabilities of choosing each column ?

Both cases could be simulated by choosing variables (partially or fully randomly) and building tree containing all variables in new temporal dataset, then repeating this procedure until we obtain assumed number of trees and then merging trees into forest,

but there are drawbacks like moving large number of data in scripting language until they are sent to low level implementation of forest for building tree or lack of merging procedures, which makes work with set of trees harder.

Answer 1

Ranger in R can do the first request. The always.split.variables argument defines which columns should always be included.

Answer 2

I know that neither xgboost, nor sklearn offer what you want. I have checked R, and didn't find it either.

But random forests are easy models to implement, so you can just produce one by yourself:

Here, in python using sklearn:

import numpy as np
from sklearn.tree import DecisionTreeClassifier
from scipy.stats import mode


class MyRandomForest:
    def __init__(self, Pcol, Pobs, n_estimators=10):
        self.n_estimators = n_estimators
        self.Pcol = Pcol  # vector
        self.Pobs = Pobs  # scalar

    def fit(self, X, y):
        self.classes_ = np.unique(y)
        assert len(self.Pcol) == X.shape[1]
        self.cols = []
        self.ms = []
        while True:
            j = np.random.rand(X.shape[1]) <= self.Pcol
            if not np.any(j):  # at least one column must be chosen!
                continue
            x = X[:, j]
            i = np.random.choice(range(len(X)), int(len(X)*self.Pobs), False)
            self.cols.append(j)
            self.ms.append(DecisionTreeClassifier().fit(x[i], y[i]))
            if len(self.ms) == self.n_estimators:
                break
        return self

    def predict(self, X):
        yp = [m.predict(X[:, cs]) for cs, m in zip(self.cols, self.ms)]
        yp = mode(yp, 0)[0][-1]
        return yp

if __name__ == '__main__':  # TEST
    from sklearn.datasets import load_iris
    from sklearn.cross_validation import StratifiedKFold
    from sklearn.metrics import accuracy_score
    iris = load_iris()
    X = iris.data
    y = iris.target
    for tr, ts in StratifiedKFold(y):
        m = MyRandomForest([0.5, 1, 0.2, 0.3], 1).fit(X[tr], y[tr])
        print(accuracy_score(y[ts], m.predict(X[ts])))

If you are using Linux, you can use multiprocessing to easily make it run in parallel.

What I did was just:

1. train a sequence of individual trees, using your probabilities
2. save both the models and whichever columns were sampled for training because we want to use only those for evaluation (or we will get an error)
3. using mode to get the most voted prediction

Note: Pobs is the fraction of observations to use. You may also want to change np.random.choice(..., ..., False) to np.random.choice(..., ..., True), or make it configurable, to allow for bootstrap with resampling. Usually, random forests are trained using Pobs=1 and resample=True.