1
$\begingroup$

I setup my neural net to use mean square error as shown below. To my understanding (and from reading the documentation) this means that if the correct result of a row is 0.7 and the net predicts 0.8 the contribution to the loss by this entry is (0.8 - 0.7) squared = 0.01

from keras.models import Sequential
from keras.layers import Dense

#...build up neural network layers here...

net.compile(optimizer = 'adam', loss = 'mean_squared_error')
net.fit(training_data, training_results, batch_size = 4, epochs = 100)

I get the following output.

Epoch 100/100 1190/1190 [==============================] - 0s 133us/step - loss: 0.0082

Wow the the loss is tiny, my little neural network is doing so well! However if I validate the result on my original training data

prediction = net.predict(training_data)
prediction_delta = (prediction - training_results)

Although some of the values in the prediction_delta are small overall the loss is way higher than 0.0082 with single values as high as 0.44. Note that this is for the same training data used to fit the net and is not the test data (which also shows similar results) so I would expect to get the value 0.082 back. How does Keras calculate this loss number?

$\endgroup$
0
$\begingroup$

When you evaluated the model, you didn't use the same way to calculate the loss as it was calculated during training.

During the training, you used mean_squared_error:

net.compile(optimizer = 'adam', loss = 'mean_squared_error')

So, you should use that same error function (mean_squared_error) while evaluating.

If you do it manually, you need this:

prediction = net.predict(training_data)
prediction_delta = np.mean((prediction - training_results) ** 2)

Or you could let the Keras do it for you automatically with evaluate() function:

net.evaluate(training_data, training_results)
$\endgroup$
  • $\begingroup$ Good suggestion, although this still doesn't quite explain the difference since if prediction_delta has a result of 0.44 the mean square error must be (significantly) bigger than 0.0082 $\endgroup$ – user67342 Feb 9 '19 at 18:52
  • $\begingroup$ But you didn't square the difference and them sum those squares and take the average? $\endgroup$ – Antonio Jurić Feb 9 '19 at 18:57
0
$\begingroup$

In this particular case, the loss is:

https://en.wikipedia.org/wiki/Mean_squared_error

i.e. the mean of the squares of the differences of your model prediction and the real value of each sample.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.