1
$\begingroup$

I tested two pieces of code and they delivered different results, which was quite unexpected.

First piece of code is supposed to train models in a k-fold manner, preserve each one of these fitted models and then validate them later on same or different dataset:

models = dict()
# train on Dataset 1
for component in components:
    print(component)
    # fetch X
    # fetch y

    kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)
    model = RandomForestClassifier(random_state=11)
    f1_scores = [[], []]
    models[component] = []
    # enumerate the splits and summarize the distributions
    for train_idx, test_idx in kfold.split(X, y):
        # select rows
        X_full_train, X_full_test = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
        # summarize train and test composition
        model.fit(X_full_train, y_train)
        models[component].append(model)

print("Dataset 1")
# evaluate on Dataset 1 samples
print()
for component in components:
    print(component)
    # fetch X
    # fetch y

    kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)
    # enumerate the splits and summarize the distributions
    predictions = []
    y_tests = []
    for train_idx, test_idx in kfold.split(X, y):
        model = models[component].pop(0)
        # select rows
        X_full_train, X_full_test = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
        # summarize train and test composition
        prediction = model.predict(X_full_test)
        predictions.extend(prediction)
        y_tests.extend(y_test)

    fig, (ax1,ax2) = plt.subplots(1,2, figsize=(9,2))
    clf_report = classification_report(y_tests,
                                       predictions,
                                       output_dict=True)
    sns.heatmap(pd.DataFrame(clf_report).iloc[:-1, :-3].T, annot=True, ax=ax1)
    ConfusionMatrixDisplay.from_predictions(y_tests, predictions, xticks_rotation=45, ax=ax2)
    plt.show()

hematophoietic 200 clf report and cm hematophoietic 270 clf report and cm liver 437 clf report and cm

Second piece of code is doing basically the same thing as the one above (in case the validation dataset is the same one as training dataset). So, I perform k-fold training and testing in one of the identically split data (because of random_state):

print("Dataset 1")
# train and evaluate on Dataset 1 samples
print()
for component in components:
    print(component)
    # fetch X
    # fetch Y

    kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)
    model = RandomForestClassifier(random_state=11)
    # enumerate the splits and summarize the distributions
    predictions = []
    y_tests = []
    for train_idx, test_idx in kfold.split(X, y):
        # select rows
        X_full_train, X_full_test = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
        # summarize train and test composition
        model.fit(X_full_train, y_train)
        prediction = model.predict(X_full_test)
        predictions.extend(prediction)
        y_tests.extend(y_test)

    fig, (ax1,ax2) = plt.subplots(1,2, figsize=(9,2))
    clf_report = classification_report(y_tests,
                                       predictions,
                                       output_dict=True)
    sns.heatmap(pd.DataFrame(clf_report).iloc[:-1, :-3].T, annot=True, ax=ax1)
    ConfusionMatrixDisplay.from_predictions(y_tests, predictions, xticks_rotation=45, ax=ax2)
    plt.show()

hematophoietic 200 clf report and cm hematophoietic 270 clf report and cm liver 437 clf report and cm

As you can see, these results look less optimistic as opposed to the first ones. What wonders me, is that they look different even though I fed them with same random_state integer and I do not quite understand why is that so? I would be glad if someone could explain this to me.

Thanks in forward!

$\endgroup$
4
  • $\begingroup$ I would suggest using the same value of random_value everywhere you can. See if it solves the problem. $\endgroup$
    – spectre
    Dec 24, 2021 at 14:27
  • $\begingroup$ @spectre sorry for the late response, I took some time off. Regarding random_state, that is basically what I did. If you take a closer look you will see that both models as well as splits have same random_state throughout two pieces. Unfortunately, same results are not delivered. $\endgroup$
    – Jumpman
    Jan 8 at 2:28
  • $\begingroup$ Your KFold and model random_state are different. $\endgroup$
    – spectre
    Jan 8 at 5:38
  • $\begingroup$ Yeah, but same troughout approaches. $\endgroup$
    – Jumpman
    Jan 11 at 15:41

1 Answer 1

1
$\begingroup$

I expected scikit to allocate completely new memory space for corresponding model during fit() call, which does not happen to be the case. So in the first case by calling

models[component].append(model)

I tend to save the address of model rather than the deep copy of the model itself. Later on, this model gets overwritten by the next one and so on. Eventually, I end up with a list of same address pointing to the last fitted model. Easy solution to this is to move the model creation inside the loop or create a deep copy manually using copy utilities:

for train_idx, test_idx in kfold.split(X, y):
    model = RandomForestClassifier(random_state=11)
    ...
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.