Random Forest Model Train, Save and Predict Later vs Train and Predict Right Away - Different Results

Question

I tested two pieces of code and they delivered different results, which was quite unexpected.

First piece of code is supposed to train models in a k-fold manner, preserve each one of these fitted models and then validate them later on same or different dataset:

models = dict()
# train on Dataset 1
for component in components:
    print(component)
    # fetch X
    # fetch y

    kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)
    model = RandomForestClassifier(random_state=11)
    f1_scores = [[], []]
    models[component] = []
    # enumerate the splits and summarize the distributions
    for train_idx, test_idx in kfold.split(X, y):
        # select rows
        X_full_train, X_full_test = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
        # summarize train and test composition
        model.fit(X_full_train, y_train)
        models[component].append(model)

print("Dataset 1")
# evaluate on Dataset 1 samples
print()
for component in components:
    print(component)
    # fetch X
    # fetch y

    kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)
    # enumerate the splits and summarize the distributions
    predictions = []
    y_tests = []
    for train_idx, test_idx in kfold.split(X, y):
        model = models[component].pop(0)
        # select rows
        X_full_train, X_full_test = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
        # summarize train and test composition
        prediction = model.predict(X_full_test)
        predictions.extend(prediction)
        y_tests.extend(y_test)

    fig, (ax1,ax2) = plt.subplots(1,2, figsize=(9,2))
    clf_report = classification_report(y_tests,
                                       predictions,
                                       output_dict=True)
    sns.heatmap(pd.DataFrame(clf_report).iloc[:-1, :-3].T, annot=True, ax=ax1)
    ConfusionMatrixDisplay.from_predictions(y_tests, predictions, xticks_rotation=45, ax=ax2)
    plt.show()

Second piece of code is doing basically the same thing as the one above (in case the validation dataset is the same one as training dataset). So, I perform k-fold training and testing in one of the identically split data (because of random_state):

print("Dataset 1")
# train and evaluate on Dataset 1 samples
print()
for component in components:
    print(component)
    # fetch X
    # fetch Y

    kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)
    model = RandomForestClassifier(random_state=11)
    # enumerate the splits and summarize the distributions
    predictions = []
    y_tests = []
    for train_idx, test_idx in kfold.split(X, y):
        # select rows
        X_full_train, X_full_test = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
        # summarize train and test composition
        model.fit(X_full_train, y_train)
        prediction = model.predict(X_full_test)
        predictions.extend(prediction)
        y_tests.extend(y_test)

    fig, (ax1,ax2) = plt.subplots(1,2, figsize=(9,2))
    clf_report = classification_report(y_tests,
                                       predictions,
                                       output_dict=True)
    sns.heatmap(pd.DataFrame(clf_report).iloc[:-1, :-3].T, annot=True, ax=ax1)
    ConfusionMatrixDisplay.from_predictions(y_tests, predictions, xticks_rotation=45, ax=ax2)
    plt.show()

As you can see, these results look less optimistic as opposed to the first ones. What wonders me, is that they look different even though I fed them with same random_state integer and I do not quite understand why is that so? I would be glad if someone could explain this to me.

Thanks in forward!

Answer 1

I expected scikit to allocate completely new memory space for corresponding model during fit() call, which does not happen to be the case. So in the first case by calling

models[component].append(model)

I tend to save the address of model rather than the deep copy of the model itself. Later on, this model gets overwritten by the next one and so on. Eventually, I end up with a list of same address pointing to the last fitted model. Easy solution to this is to move the model creation inside the loop or create a deep copy manually using copy utilities:

for train_idx, test_idx in kfold.split(X, y):
    model = RandomForestClassifier(random_state=11)
    ...