Am I using GridSearch correctly or do I need to use all data for cross validation?

Question

I'm working with a dataset that has 400 observations, 34 features and quite a few outliers, some of them extreme. Given the nature of my data, these need to be in the model.

I started by doing a 75-25 split on my data and leaving those 25% aside.

With the train set, I used GridSearchCV with a RepeatedKFold of 10 folds and 7 repeats and this returned my best_estimator results, which when we go in .cv_results_ we see it's the mean_test_score metric. I then called this my "Cross Validation score". Then, with this model fit, I ran it on the test set as grid.score(X_test, y_test) and called this my Test score.

def rf(df, score):

    X_train, X_test, y_train, y_test = train_test(df)

    params = {'n_estimators': [400, 700, 1000],
              'max_features': ['sqrt', 'auto'],
              'min_samples_split': [2, 3],
              'min_samples_leaf': [1, 2, 3],
              'max_depth': [50, 100, None],
              'bootstrap': [True, False]
}

    scorers = {'RMSE': make_scorer(rmse, greater_is_better=False),
               'MAE': make_scorer(mean_absolute_error, greater_is_better=False),
               'R2': make_scorer(r2_score)}

    cv = RepeatedKFold(n_splits=10, n_repeats=7)


    grid = GridSearchCV(estimator=RandomForestRegressor(random_state=random.seed(42)),
                              param_grid=params, 
                              verbose=1, 
                              cv=cv, 
                              n_jobs =-1, 
                              scoring=scorers, 
                              refit = score)

    grid = grid.fit(X_train, y_train)    

    print('Parameters used:', grid.best_params_)

    if score  == 'RMSE':
        print('RMSE score on train:', round(-1*grid.best_score_,4))
        print('RMSE score on test: ', round(-1*grid.score(X_test, y_test),4))

    elif score == 'R2':
        print('R Squared score on train:', round(grid.best_score_,4))
        print('R Squared score on test: ', round(grid.score(X_test, y_test),4))

    elif score == 'MAE':
        print('MAE score on train:', round(-1*grid.best_score_,4))
        print('MAE score on test: ', round(-1*grid.score(X_test, y_test),4))

When I set my metric to RMSE (the most important one), this is what it outputs:

RMSE score on train: 8.489
RMSE score on test: 5.7952

Have I done this correctly? Can I consider this discrepancy acceptable? With Random Forest for example, if I deliberately ignore the gridsearch parameters and set my min_leaf_node to something like 10, my RMSE goes all the way up to 12 but it becomes very similar between the CV score and my test data. I'm experiencing similar results with SVR and MLP algorithms.

This is part of my thesis and now I have my supervisor telling me I should be using all my data for cross-validation which I don't think is correct.

My conclusion is that given the outliers in the model, without more observations, a discrepancy in results is to be expected, however I don't know if this conclusion is right or if I'm doing something wrong here.

Running my model in a somewhat similar dataset with fewer outliers gives results closer to one another.

RMSE score on train: 5.9731
RMSE score on test: 6.9164

Answer 1

Your procedure is, from what I can tell, correct. You are correctly splitting your data into train/test, and then using your training data only to find optimal hyper-parameters. Using all of the training data and the hyper parameters found in cross validation, you are then evaluating your final model on the test set.

Indeed, the outliers and the size of your dataset are the most probable causes for the large differences between validation and test. Basically, if the majority of these outlier observations fall in the test set after your initial splitting of your data it is highly likely that your test set scores will be larger than your validation scores. On the other hand, if these outliers are in your training set, you would expect the opposite to be true. This is because, regardless of what data your have in your training set for any random partitioning, these outlier observations are unlikely to be predicted well regardless of what you do, and because the size of your dataset is so small, these outliers will heavily impact your error estimate (especially with RMSE that is incredibly sensitive to outliers due to the squaring).

Either way, report the test set scores only as your final estimates if you want to be honest.

In scenarios in which the variance is large in my test set scores (usually due to a small data set and a large number of outliers), I would highly recommend repeating your entire model building procedure as you described but with a different seed each time (i.e. split your entire dataset into train/test with a different seed each time). Do this until you run out of patience or the variance in your estimate of model performance is small enough. Maybe form bootstrapped confidence intervals from these repetitions to give you information on how much your model's performance varies.