Model performance worsens after Cross Validation

Question

I am training a logistic regression model on a dataset with only numerical features. I performed the following steps:-

1.) heatmap to remove collinearity between variables

2.) scaling using StandarScaler

3.) cross validation after splitting, for my baseline model

4.) fitting and predicting

Below is my code:-

# SPLITTING 
train_x, test_x, train_y, test_y = train_test_split(data2, y, test_size = 0.2, random_state = 
69)

#MODEL INSTANCE
model = LogisticRegression(random_state = 69)

# SCALING
train_x2 = train_x.copy(deep = True)
test_x2 = test_x.copy(deep = True)

s_scaler = StandardScaler()
s_scaler.fit(train_x2)
s_scaled_train = s_scaler.transform(train_x2)
s_scaled_test = s_scaler.transform(test_x2)

# BASELINE MODEL
cross_val_model2 = -1 * cross_val_score(model, s_scaled_train, train_y, cv = 5,
                              n_jobs = -1, scoring = 'neg_mean_squared_error')
s_score = cross_val_model2.mean()

# FITTING AND PREDICTING
model.fit(s_scaled_train, train_y)
pred = model.predict(s_scaled_test)
mse = mean_squared_error(test_y, pred)

CV score is 0.06 and score after fitting and predicting is 0.23. I find this weird as CV is a measure of how good your model performs. So I should atleast get a score equal to the CV score right?

Answer 1

A couple additional points:

• With MSE or any error measure, the value is relative to the data. This implies that one cannot know what the difference between 0.06 and 0.23 represents: this difference might be very large or very small, it all depends on the distribution of the target.
• As Sammy mentions, the small size of the data is likely to cause high variations depending on the split. In order to understand the difference it would be useful to observe the performance variation across CV folds, for example by calculating the standard deviation: if the variation is high then the model is not stable, and that could explain the high difference.

Answer 2

With only 210 samples the difference might be caused by your train and test data not being from the same underlying distribution. That is, using holdout-CV to estimate model performance on such a small dataset can be susceptible to how you split your data. You can test this by re-running the pipeline with a different train/test-split and see if results differ.

Instead, for small datasets I suggest to apply nested k-fold CV. You can read more about in, for example, Bias in Error Estimation When Using Cross-Validation for Model Selection. Here you can find an implementation in scikit learn.

Answer 3

No expert here, so dont take my words too close to heart, but I can think of 2 reasons for this. First, the CV, if split into too many sections, can not work prperly (depending on the data size basically), even cause overfitting. Second, The metric you are using: mean_squared_error will not necessarily show you the results that most of us want (I dont know if thats the case with you). Try using R2 score, might give you different results. Good luck! hope this helps even a tiny bit