# Random Forest Model Train, Save and Predict Later vs Train and Predict Right Away - Different Results

I tested two pieces of code and they delivered different results, which was quite unexpected.

First piece of code is supposed to train models in a k-fold manner, preserve each one of these fitted models and then validate them later on same or different dataset:

models = dict()
# train on Dataset 1
for component in components:
print(component)
# fetch X
# fetch y

kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)
model = RandomForestClassifier(random_state=11)
f1_scores = [[], []]
models[component] = []
# enumerate the splits and summarize the distributions
for train_idx, test_idx in kfold.split(X, y):
# select rows
X_full_train, X_full_test = X.iloc[train_idx], X.iloc[test_idx]
y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
# summarize train and test composition
model.fit(X_full_train, y_train)
models[component].append(model)

print("Dataset 1")
# evaluate on Dataset 1 samples
print()
for component in components:
print(component)
# fetch X
# fetch y

kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)
# enumerate the splits and summarize the distributions
predictions = []
y_tests = []
for train_idx, test_idx in kfold.split(X, y):
model = models[component].pop(0)
# select rows
X_full_train, X_full_test = X.iloc[train_idx], X.iloc[test_idx]
y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
# summarize train and test composition
prediction = model.predict(X_full_test)
predictions.extend(prediction)
y_tests.extend(y_test)

fig, (ax1,ax2) = plt.subplots(1,2, figsize=(9,2))
clf_report = classification_report(y_tests,
predictions,
output_dict=True)
sns.heatmap(pd.DataFrame(clf_report).iloc[:-1, :-3].T, annot=True, ax=ax1)
ConfusionMatrixDisplay.from_predictions(y_tests, predictions, xticks_rotation=45, ax=ax2)
plt.show()


Second piece of code is doing basically the same thing as the one above (in case the validation dataset is the same one as training dataset). So, I perform k-fold training and testing in one of the identically split data (because of random_state):

print("Dataset 1")
# train and evaluate on Dataset 1 samples
print()
for component in components:
print(component)
# fetch X
# fetch Y

kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)
model = RandomForestClassifier(random_state=11)
# enumerate the splits and summarize the distributions
predictions = []
y_tests = []
for train_idx, test_idx in kfold.split(X, y):
# select rows
X_full_train, X_full_test = X.iloc[train_idx], X.iloc[test_idx]
y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
# summarize train and test composition
model.fit(X_full_train, y_train)
prediction = model.predict(X_full_test)
predictions.extend(prediction)
y_tests.extend(y_test)

fig, (ax1,ax2) = plt.subplots(1,2, figsize=(9,2))
clf_report = classification_report(y_tests,
predictions,
output_dict=True)
sns.heatmap(pd.DataFrame(clf_report).iloc[:-1, :-3].T, annot=True, ax=ax1)
ConfusionMatrixDisplay.from_predictions(y_tests, predictions, xticks_rotation=45, ax=ax2)
plt.show()


As you can see, these results look less optimistic as opposed to the first ones. What wonders me, is that they look different even though I fed them with same random_state integer and I do not quite understand why is that so? I would be glad if someone could explain this to me.

Thanks in forward!

• I would suggest using the same value of random_value everywhere you can. See if it solves the problem. Dec 24, 2021 at 14:27
• @spectre sorry for the late response, I took some time off. Regarding random_state, that is basically what I did. If you take a closer look you will see that both models as well as splits have same random_state throughout two pieces. Unfortunately, same results are not delivered. Jan 8 at 2:28
• Your KFold and model random_state are different. Jan 8 at 5:38
• Yeah, but same troughout approaches. Jan 11 at 15:41

models[component].append(model)

for train_idx, test_idx in kfold.split(X, y):