# Why is doing a ShuffleSplit with GridSearchCV decreasing the performance of my model?

I'm using the Boston Housing dataset in Keras, and neural networks for this regression problem.

The following is the code I use to prepare the data, build the model, and fit it with GridSearchCV.

from keras import models
from keras import layers
import numpy as np
from sklearn.model_selection import GridSearchCV
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import ShuffleSplit

from keras.datasets import boston_housing

mean=np.mean(train_data)
std=np.std(train_data)

train_data_norm=(train_data-mean)/std
test_data_norm=(test_data-mean)/std

def build_model():
model=models.Sequential()
input_shape=(train_data_norm.shape[1],)))
model.compile(optimizer='rmsprop',loss="mse",metrics=["mae"])
return model

model=KerasRegressor(build_fn=build_model,epochs=30,verbose=0)

param_grid = {"epochs":range(1,11)}

grid_model=GridSearchCV(model,param_grid,cv=4,n_jobs=-1,scoring='neg_mean_squared_error')


When I run

grid_model.fit(train_data, train_targets)

mean_squared_error(grid_model.predict(test_data),test_targets)


several times, the performance seemed to vary a lot, with the MSE ranging from 60 to 90.

So, I thought that if I used ShuffleSplit, then maybe I could stabilise the scoring... For that, I've used this code:

ss = ShuffleSplit(n_splits=4, test_size=0.1, random_state=0)

grid_model=GridSearchCV(model,param_grid,cv=ss,n_jobs=-1,scoring='neg_mean_squared_error')

grid_model.fit(train_data, train_targets)

mean_squared_error(grid_model.predict(test_data),test_targets)


However, now the MSE is even bigger! I've ran the last statement several times and I get an MSE ranging from 110 to 190. Not only it didn't stabilise, it also got worse. Why is that? Is my code incorrect somehow?

• Try increasing the test size on the suffle split, since this is only .1 the variance of the estimates will be greater than the one that you see when running cv (default is 5 fold so your test size is 1/5 * X_train.shape[0] > than .1*X_train.shape[0] – Julio Jesus Mar 1 at 22:39
• @JulioJesus thanks for the comment. I've tried that, but even with 0.20 or 0.5, the values were still of the same magnitude... – An old man in the sea. Mar 2 at 0:10
• You might find this interesting: stats.stackexchange.com/questions/51416/… – Julio Jesus Mar 2 at 2:05
• Finally I would like to add that, unless you are using a fixed seed on both cv method and model itself, another random factor will be added making difficult to get comparable results – Julio Jesus Mar 2 at 2:09
• @JulioJesus and thanks for the link ;) – An old man in the sea. Mar 2 at 10:34