I'm using the Boston Housing dataset in Keras, and neural networks for this regression problem.
The following is the code I use to prepare the data, build the model, and fit it with GridSearchCV.
from keras import models
from keras import layers
import numpy as np
from sklearn.model_selection import GridSearchCV
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import ShuffleSplit
from keras.datasets import boston_housing
(train_data,train_targets),(test_data,test_targets)=boston_housing.load_data()
mean=np.mean(train_data)
std=np.std(train_data)
train_data_norm=(train_data-mean)/std
test_data_norm=(test_data-mean)/std
def build_model():
model=models.Sequential()
model.add(layers.Dense(64,activation="relu",
input_shape=(train_data_norm.shape[1],)))
model.add(layers.Dense(64,activation="relu"))
model.add(layers.Dense(1))
model.compile(optimizer='rmsprop',loss="mse",metrics=["mae"])
return model
model=KerasRegressor(build_fn=build_model,epochs=30,verbose=0)
param_grid = {"epochs":range(1,11)}
grid_model=GridSearchCV(model,param_grid,cv=4,n_jobs=-1,scoring='neg_mean_squared_error')
When I run
grid_model.fit(train_data, train_targets)
mean_squared_error(grid_model.predict(test_data),test_targets)
several times, the performance seemed to vary a lot, with the MSE ranging from 60 to 90.
So, I thought that if I used ShuffleSplit, then maybe I could stabilise the scoring... For that, I've used this code:
ss = ShuffleSplit(n_splits=4, test_size=0.1, random_state=0)
grid_model=GridSearchCV(model,param_grid,cv=ss,n_jobs=-1,scoring='neg_mean_squared_error')
grid_model.fit(train_data, train_targets)
mean_squared_error(grid_model.predict(test_data),test_targets)
However, now the MSE is even bigger! I've ran the last statement several times and I get an MSE ranging from 110 to 190. Not only it didn't stabilise, it also got worse. Why is that? Is my code incorrect somehow?
