# Is fitting two RandomForestClassifiers 500 trees each and average their predicted probabilities on the test set more performant than one with 1000?

If I fit two RandomForestClassifiers 500 trees each and average their predicted probabilities on the test set, would it have better results than fitting a RandomForestClassifier with 1000 trees and use it to get test set probabilities?

As these algorithms are random based I would say that their performance should be roughly the same?

I am okay with some math to prove it, or any other way that might prove it.

• Have you tried with a toy dataset? Oct 5 '20 at 18:51

With a toy dataset, I obtained slightly better results with the RF 1000 trees

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

from scipy.stats import ks_2samp

import matplotlib.pyplot as plt
plt.style.use("seaborn-whitegrid")

n_models = 2
threshold = .5

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 42, test_size = .3)

rForest1000 = RandomForestClassifier(n_estimators= 1000, random_state= 42, oob_score= True).fit(X_train, y_train)
preds1000 = rForest1000.predict_proba(X_test)[:,1]

roc_score = roc_auc_score(y_true = y_test, y_score= preds1000)

print(f"Test set score for 1K trees is : {round(roc_score, 4)}")

ls = list()
for i in range(n_models):
model = RandomForestClassifier(n_estimators= 500, random_state= i).fit(X_train, y_train)
ls.append(model.predict_proba(X_test)[:,1])
preds = np.array(ls).mean(axis =0)

roc_score = roc_auc_score(y_true = y_test, y_score= preds)

print(f"Test set score for 500trees X 2 avg is : {round(roc_score, 4)}")

fig, ax = plt.subplots(1,2, figsize = (12,5))

ax[0].hist(preds1000[y_test == 0], color = "darkgreen", alpha = .5)
ax[0].hist(preds1000[y_test == 1], color = "darkred", alpha = .5)

ax[0].set_title(f"Predictions distribution with 1K Trees RF\n KS: {[round(ks_2samp(preds1000[y_test == 0], preds1000[y_test == 1])[0],3)]}")

ax[1].hist(preds[y_test == 0], color = "darkgreen", alpha = .5)
ax[1].hist(preds[y_test == 1], color = "darkred", alpha = .5)

ax[1].set_title(f"Predictions distribution with X2 500 Trees RF Trees RF\n KS: {[round(ks_2samp(preds[y_test == 0], preds[y_test == 1])[0],3)]}");

• If you rerun this with different random seeds and analyze the resulting score distributions, I think you'll find there is no statistically significant difference. Oct 5 '20 at 19:57
• @BenReiniger I completely agree with you. Nonetheless, I have seen the second approach of averaging RF predictions on Kaggle competitions, in my opinion at the end is a matter of experimentation especially when a .001% improvement on performance is the difference between the 1st and the 11th place on a Kaggle contest, but in practical terms, there is no difference Oct 5 '20 at 20:23
• If the two RF's have different hyperparameters, then it's a different story (and that includes kaggle "ensemble-all-the-public-models-together" types). But if they have the same hyperparameters, then it's equivalent to the more-tree version, up to the random effects: you can't know in advance which one will come up on top for an unseen dataset. Oct 6 '20 at 14:26