0
$\begingroup$

I am encountering a problem where the validation error I see on the learning curve of my machine learning model is different from the validation error I calculate using the mean squared error function. I am not sure if this is the expected behavior or if there is an error in my code. I am seeking advice on how to interpret and address this issue.

Here is my Random Forest model:

    # Define the model
rf_model = RandomForestRegressor(random_state=42)

# Define the parameter grid for pruning
param_grid = {
    'max_depth': [5, 10, 15],
    'min_samples_split': [2, 5, 10],
    'min_samples_leaf': [1, 2, 3, 4],
}

# Perform a grid search to find the best hyperparameters
grid_search = GridSearchCV(estimator=rf_model, param_grid=param_grid, cv=5)
grid_search.fit(X_train, y_train)
best_rf_reg = grid_search.best_estimator_

# Train the model on the training set using the best hyperparameters
best_rf_reg.fit(X_train, y_train)

# Make predictions on the test set
y_pred = best_rf_reg.predict(X_test)

rf_r2 = r2_score(y_test, y_pred)
rf_mse = mean_squared_error(y_test, y_pred)
rf_mae = mean_absolute_error(y_test, y_pred)

print("Rˆ2 in random forest:", rf_r2)
print("MSE in random forest:", rf_mse)
print("MAE in random forest:", rf_mae)

And it's output:

  • Rˆ2 in random forest: 0.788561260837054
  • MSE in random forest: 171727.5491440435
  • MAE in random forest: 284.4151697399289

And I use this code to get the validation and train errors for learning curve:

from sklearn.model_selection import learning_curve
import matplotlib.pyplot as plt

# Replace 'model' with the name of your model
train_sizes, train_scores, test_scores = learning_curve(best_rf_reg, X, y, cv=5, scoring = 'neg_mean_squared_error', shuffle = True)

train_scores_mean = -train_scores.mean(axis=1)
validation_scores_mean = -test_scores.mean(axis=1)
print('Mean training scores\n\n', pd.Series(train_scores_mean, index=train_sizes))
print('\n', '-' * 20)  # separator
print('\nMean validation scores\n\n', pd.Series(validation_scores_mean, index=train_sizes))

Mean training scores:

  • 93050.572113
  • 121290.780987
  • 135467.911224
  • 145831.052914
  • 149380.086868

Mean validation scores:

  • 274654.971506
  • 226675.486281
  • 214613.725089
  • 212955.842776
  • 212945.602182

Why in this output validation scores significantly higher that output from the model?

enter image description here

New contributor
Omarov Alen is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$

0

Your Answer

Omarov Alen is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy