Tuned model has higher CV accuracy, but a lower test accuracy. Should I use the tuned or untuned model?

Question

I am working on a classification problem using Sci Kit Learn and am confused on how to properly tune hyper parameters to get the "best" model.

Before any tuning, my logistic regression classifier has a 74.6% accuracy on the test set.

To choose the optimal parameters for my final model, I fit a GridSearchCV object to my training data with a parameter grid that included the default parameters of the LogisticRegression classifier from Sci Kit Learn.

The CV accuracy from the GridSearchCV object I fit was 76.5% which would suggest this model will have a higher accuracy than the untuned model.

When I fit and evaluate the tuned model on the test set, I get an accuracy of 73%.

^ This is the part that is confusing to me. I know that the results from CV are supposed to estimate how well the model will perform, but it actually lowered the test accuracy.

Does this mean that I should proceed with the "best" model found via GridSearchCV, or should I use the untuned model because it had a higher accuracy on the test set ?

Code used (can't provide the data)

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

# Split the data
X = data.drop(columns=target)
y = data[target]
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=10)


# Define different Pipelines for Numeric vs. Categorical
numeric_transformer = Pipeline(
    steps=[
        ("numeric_imputer", SimpleImputer(strategy="mean")),
        ("scaler", StandardScaler())
        ]
)

categorical_transformer = Pipeline(
    steps=[
        ("categorical_imputer", SimpleImputer(strategy="most_frequent")),
        ("ohe", OneHotEncoder(handle_unknown="ignore"))
        ]
)

numeric_features = X_train.select_dtypes(include="number").columns.values
categorical_features = X_train.select_dtypes(exclude="number").columns.values

# Combine into single preprocessor
preprocessor = ColumnTransformer(
    transformers=[
        ("cat", categorical_transformer, categorical_features),
        ("num", numeric_transformer, numeric_features),
    ]
)


# Score the model before tuning parameters
steps = [
        ('preprocessor', preprocessor),
        ('regressor', LogisticRegression())
    ]
pipeline = Pipeline(steps)
pipeline.fit(X_train, y_train)
accuracy = pipeline.score(X_test, y_test)
print(f"Test Accuracy before Hyperparamter Tuning\n{accuracy:,}")

##############################################################################

# Model Tuning
steps = [
        ('preprocessor', preprocessor),
        ('regressor', LogisticRegression())
    ]
pipeline = Pipeline(steps)

# Set up the parameter grid to search over
param_grid = {
    "regressor__solver": ['newton-cg', 'lbfgs', 'liblinear'],
    "regressor__penalty": ['l1', 'l2', 'elasticnet'],
    "regressor__C": [100, 10, 1.0, 0.1, 0.01],
    "regressor__fit_intercept": [False, True]
}

cv = GridSearchCV(pipeline
                  , cv = 4
                  , param_grid=param_grid
                 , scoring='accuracy')

cv.fit(X_train, y_train)
accuracy = cv.score(X_train, y_train)
print(f"Hyperparamter Tuned Model CV Accuracy\n{accuracy:,}")
print(f"Tuned Model Best Params: {cv.best_params_}")

# Final Test of Tuned model
cv.best_estimator_.fit(X_train, y_train)
accuracy = cv.best_estimator_.score(X_test, y_test)
print(f"Test Accuracy before Hyperparamter Tuning\n{accuracy:,}")

Result of above code

EDIT:

I found if I fit the GridSearchCV object on the entire data set (instead of training data only), then the "best" model returned from the GridSearchCV object is indeed a LogisticRegression classifier with default parameters.

I recall reading that CV should be applied to the training set only to select a model so that you can use the test set as the final validation for the selected model.

^ Is this the correct approach, or should I be using CV on all of the data to select my model ?

I know that under the hood CV splits the data into train and test sets already, but I thought you still wanted to leave a final test set outside of this process ?

Answer 1

The k-fold cross-validation (CV) used for Grid Search in sklearn indeed divides the dataset that is provided to the Grid Search into k (usually 5, which is the default) folds. In your case k = 4, since you specified cv = 4.

In the training process, each of these 4 folds is once used as a validation set (which is different to a test set), and the rest of the time it is part of the training set. The best model according to the cross-validation procedure in the grid search is then selected and trained on all 4 folds to arrive at the final trained model.

Since all the data provided to the grid search is used as training data for the model and for hyperparameter tuning, one cannot unbiasedly evaluate the model's performance using any of this data. Instead, a hold-out test set that has neither been used to tune hyperparameters, nor to train the model, is required. Using this test set, the final, unbiased evaluation of the model's accuracy can be performed.

In your case, the accuracy of 0.7306667 is the correct estimate of the model's accuracy, while 0.7653334 is the accuracy on the training set and does not reflect whether it can also be achieved on new, unseen data. So stick with your approach of running GridSearchCV on X_train and y_train, then calculating accuracy of the trained model on X_test and y_test.