2
$\begingroup$

Can someone give me a tip on how I could incorporate MSE & loss plots? I have been following some machinelearningmastery posts to plot this but the application is classification and I am attempting regression. Also what is different in my script is I am defining the model thru calling a function, so I am curious if my script could be re-written without the function def wider_model() that defines the model.

This script below works except what is commented out on the bottom for the plt plots. In the machinelearningmastery post, someone does ask this question how to do for regression, and supposedly if you print print(history.history.keys()) two values are returned for dict_keys([‘mean_absolute_error’, ‘loss’])...

Any tips help, there isn't a lot of wisdom here... Thank you

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import matplotlib.pyplot as plt
import math



# load dataset
dataset = pandas.read_csv("prepdSPdata.csv", index_col='Date', parse_dates=True)

# shuffle dataset
dataset = dataset.sample(frac=1.0)


# split into input (X) and output (Y) variables
X = numpy.array(dataset.drop(['Demand'],1))
Y = numpy.array(dataset['Demand'])

print(dataset.shape)
print(dataset.dtypes)
print(dataset.columns)


def wider_model():
    # create model
    model = Sequential()
    model.add(Dense(20, input_dim=11, kernel_initializer='normal', activation='relu'))
    model.add(Dense(10, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model


# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=wider_model, epochs=1, batch_size=5, verbose=0)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Wider: %.2f (%.2f) MSE" % (results.mean(), results.std()))
print("RMSE", math.sqrt(results.std()))



# list all data in history
#print(wider_model.wider_model.keys())

# summarize history for MSE
#plt.plot(history.history['acc'])
#plt.plot(history.history['val_acc'])
#plt.title('model MSE')
#plt.ylabel('MSE')
#plt.xlabel('epoch')
#plt.legend(['train', 'test'], loc='upper left')
#plt.show()

# summarize history for loss
#plt.plot(history.history['loss'])
#plt.plot(history.history['val_loss'])
#plt.title('model loss')
#plt.ylabel('loss')
#plt.xlabel('epoch')
#plt.legend(['train', 'test'], loc='upper left')
#plt.show()
$\endgroup$
2
$\begingroup$

cross_val_score does not return the history of the training. You can use fit instead:

history = model.fit( ...

See this example.

As you mentioned, the history object holds the results of the training for each epoch.

Here is the relevant bit:

history = model.fit(X, X, epochs=500, batch_size=len(X), verbose=2)
pyplot.plot(history.history['mean_squared_error'])
pyplot.plot(history.history['mean_absolute_error'])
pyplot.plot(history.history['mean_absolute_percentage_error'])
pyplot.plot(history.history['cosine_proximity'])
pyplot.show()

I was actually working on the same example that you referenced yesterday. I think it is hard to understand because it introduces many functions and concepts: estimators, StandardScaler, KerasRegressor, Pipeline, KFold and cross_val_score.

However, I did like the approach to creating and testing models, and Cross Validation would produce more robust models.

$\endgroup$
1
$\begingroup$

You're only training your model for 1 epoch so you're only giving it one data point to work from. If you want to plot a line of loss or accuracy you need to train for more epochs.

$\endgroup$
5
  • $\begingroup$ Sorry that was jus testing my code I am training for about 100 epochs $\endgroup$
    – HenryHub
    Feb 21 '19 at 17:40
  • $\begingroup$ Ah okay, could you give more explanation as to why the commented code doesn’t work? $\endgroup$
    – HFulcher
    Feb 21 '19 at 17:41
  • 1
    $\begingroup$ cross_val_score does not return the history of the training. You can use fit instad: history = model.fit( ... $\endgroup$
    – B Seven
    Feb 21 '19 at 21:25
  • $\begingroup$ Here is an example: machinelearningmastery.com/… $\endgroup$
    – B Seven
    Feb 21 '19 at 21:27
  • $\begingroup$ @BSeven submit this as an answer! 😊 $\endgroup$
    – HFulcher
    Feb 21 '19 at 22:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.