0
$\begingroup$

I have this random forest model setup as shown below in python. It's performing unexpectedly well with a ~70% classification success rate (to the extent where I really doubt it is genuine) and I am therefore skeptical that I haven't accidentally fed it some training data - but I can't find any evidence of this.

So, I have two questions:

  • Have I made an error somewhere in this model?
  • How can I be more certain that I have not accidentally made the model predict on some training data?

Code:

##################################
###### Set up forest model #######
##################################

X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X_concatenated, y, test_size=0.2)

print(f'X_train len: {X_train}')
print(f'X_test len: {X_test}')
print(f'y_train len: {y_train}')
print(f'y_test len: {y_test}')

# Define the model
rfmodel = RandomForestClassifier(n_estimators=100)

# Define the hyperparameters to optimize
param_distributions = {
    'n_estimators': randint(10, 1000),
    'max_depth': randint(2, 50),
    'min_samples_split': randint(2, 10),
    'min_samples_leaf': randint(1, 10),
    'max_features': ['sqrt', 'log2'],
    'criterion': ['gini', 'entropy'],
    'bootstrap': [True, False],
    'class_weight': [None, 'balanced', 'balanced_subsample']
}

# Define the search strategy
search = RandomizedSearchCV(
    rfmodel,
    param_distributions=param_distributions,
    n_iter=optruncount,
    cv=5,
    random_state=42,
    n_jobs=-1
)

# Train the model with hyperparameter optimization
search.fit(X_train, y_train)

# Get the best hyperparameters
best_params = search.best_params_

# Train the final model with the best hyperparameters
rfmodel = RandomForestClassifier(**best_params)
rfmodel.fit(X_concatenated, y)

################################
###### Test forest model #######
################################

predicted = rfmodel.predict(X_test)

cm = confusion_matrix(y_test, predicted)
sns.heatmap(cm, annot=True, cmap='Blues', fmt='g')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.show()
```
$\endgroup$

1 Answer 1

0
$\begingroup$

You are training a random forest with the best hyperparameters you've found on the (X_concatenated, y) dataset, then testing it's performance using the (X_test, y_test) dataset. However, the second dataset is a subsample of the first one (see your call to train_test_split), and as a result, observations from your 'test' set will be present in 'training' set. This is data leakage and the performance metrics on your 'test' set will overestimate the actual performance of your model.

$\endgroup$
2
  • $\begingroup$ Thank you, I knew it was too good to be true. Am I correct in thinking the correction for this would be to change rfmodel.fit(X_concatenated, y) to rfmodel.fit(X_train, y_train)? $\endgroup$
    – weggegon
    Feb 17, 2023 at 14:22
  • $\begingroup$ That would indeed be correct. $\endgroup$
    – Oxbowerce
    Feb 17, 2023 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.