-1
$\begingroup$

This example is taken from the book Deep Learning With Python from Jason Brownlee. It applies a fully connected neural model with one hidden layer (13, 13, 1) using Keras library and the Tensorflow backend.

1 - Import the packages

import numpy
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_boston

2 - Load the dataset

boston = load_boston()
X = boston.data[:,0:13]
Y = boston.target

3 - Define base model

def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(13, input_dim=13, \
                    kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

4 - Fix random seed for reproducibility

seed = 7
numpy.random.seed(seed)

5 - Fit & evaluate model

estimator = KerasRegressor(build_fn=baseline_model, epochs=10, \ 
                           batch_size=5, verbose=0)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(estimator, X, Y, cv=kfold)

6 - Value of results parameter

Out[63]: 
array([-13.89524042, -14.2215869 ,  -6.21156782, -42.65242339,
       -26.58890147, -56.30860755, -28.6575911 , -89.67339525,
       -27.7172946 , -22.67604859])

Which returns a mean negative value

print("Baseline: %.2f (%.2f) MSE" % (results.mean(), results.std()))

It returns about -30 and it should return about +30 according to the book. I've tried both Theano and Tensorflow with no success.

I've also tested this code both on Windows and Linux, having obtained the same bad result.

The problem seems to be in the cross-validation part, because if I don't run cross validation, I get more sensible results.

7 - Without cross-validation

model = Sequential()
model.add(Dense(13, input_dim=13, kernel_initializer='normal', \ 
                activation='relu'))
model.add(Dense(1, kernel_initializer='normal')) 
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(X, Y, epochs=150, batch_size=10)

Now if I evaluate the model I get a more sensible value

In [68]: model.evaluate(X, Y)
506/506 [==============================] - 0s 58us/step
Out[68]: 27.778296670423664

What can be happening here? Why is the cross-validation procedure returning negative values?

$\endgroup$
0
$\begingroup$

"This is to be expected. sklearn has changed their API to invert their cost functions. Nothing to be concerned about."

I got this answer from Dr. Jason Brownlee.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.