1
$\begingroup$

My earlier question has unfortunately not been answered satisfactorily (as shown below if an epoch consists of a single batch the same strange behavior is observed), and this issue is driving me crazy, so please forgive the reposting of the question with new, more targeted, very simple code. Here is the link to my previous version of essentially the same question:

Seems like the Keras .fit and .evaluate methods give different training accuracy (yet the same validation accuracy). Same thing on loss.?

(I tried to comment to the proposed answer to that topic, but the allowed space is too short to enable posting of the new code).

Note that in the new example below an epoch has a single batch (2 examples) and the code is trained for a single epoch, so 1 epoch = 1 batch. Therefore what model.fit prints should match what model.evaluate prints. (This has been motivated by Andrej Karpathy's recommendation to start with fitting a single batch)

import numpy as np
np.random.seed(42)
import keras
from keras import layers
from keras.regularizers import l2
import tensorflow as tf
tf.random.set_seed(42)
import keras.backend as K

#def mse_custom(y_true, y_pred): #Tried defining the loss, got same results           
#    custom = K.mean(K.square((y_pred - y_true)))    
#    return custom

#Below I use 20 examples because the issue becomes much clearer than with a few examples
x_train = np.array([0.10526316, 0. ,0.26315789,0.73684211,0.94736842,1., \
                    0.52631579,0.84210526,0.21052632, 0.63157895,0.42105263,0.31578947,\
                    0.36842105,0.78947368,0.15789474,0.68421053, 0.47368421,0.05263158,\
                    0.57894737,0.89473684]).reshape(20,1,1,1)
y_train = np.array([0.31438135, 0. ,0.48843161,0.83670357,0.86771850,1., \
                    0.70921682,0.85732503,0.48475008,0.79387635,0.66044467,0.46540228,\
                    0.57612215,0.97176724,0.43534982,0.83881293,0.61425264,0.30733072,\
                    0.68508229,0.87707974]).reshape(20,1)
input_shape = (None,None,1)  #(1,1,1) gives the same results

seedbase = 1
model = keras.Sequential(
    [
        keras.Input(shape=input_shape),
        layers.Dense(100, activation="sigmoid", use_bias=True,
                     kernel_initializer=keras.initializers.RandomNormal(mean=0., stddev=1./np.sqrt(1),seed=seedbase+0), #/np.sqrt(30)
                     bias_initializer=keras.initializers.RandomNormal(mean=0., stddev=1.,seed=seedbase+1),
                     kernel_regularizer=l2(0),
                     bias_regularizer=None),
        layers.Dense(1, activation="sigmoid", use_bias=True,
                     kernel_initializer=keras.initializers.RandomNormal(mean=0., stddev=1./np.sqrt(1),seed=seedbase+12), #/np.sqrt(10)
                     bias_initializer=keras.initializers.RandomNormal(mean=0., stddev=1.,seed=seedbase+13), 
                     kernel_regularizer=l2(0),
                     bias_regularizer=None),
    ]
)

model.summary()

"""
## Train the model
"""
n_val = 2
n_tr =20-n_val
val_split = n_val/20
batch_size = 2
epochs = 1
opt = keras.optimizers.SGD(learning_rate=0.1)
#opt='adam'
#opt = keras.optimizers.RMSprop(0.00099)
model.compile(loss='mean_squared_error', optimizer=opt)
ann=model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=val_split, verbose = 1)
"""
## Evaluate the trained model
"""
print("\n")
score = model.evaluate(x_train[:n_tr], y_train[:n_tr], batch_size=1, verbose=0)
print("Training loss:", score)
score = model.evaluate(x_train[-n_val:], y_train[-n_val:], batch_size=1, verbose=0)
print("Validation loss:", score)

And if you increase the epochs to say 100, the model.fit training loss goes to 0 (exactly matching only one of the two examples but very far from the other example, so clearly incorrect, and this is a clue to what is wrong) and training stops. The model.evaluate training loss stabilizes at the correct value. And the validation model.fit loss is very close to the validation model.evaluate loss.

Also, if I increase the number of examples in a single batch, the mismatch becomes much smaller.

I know I must be doing something wrong, but what? Your help would be very much appreciated.

$\endgroup$
3
  • $\begingroup$ Doing a quick check it seems that the batch_size changes the reported score when using model.evaluate(), which seems really weird to me. $\endgroup$
    – Oxbowerce
    Apr 21 at 18:13
  • 1
    $\begingroup$ For " 1 epoch = 1 batch", you should set batch_size=20 not 2. Doing so will give you a consistent result. $\endgroup$
    – 10xAI
    Apr 22 at 7:39
  • $\begingroup$ @Oxbowerce This is what I explained in the answer to the linked question. $\endgroup$
    – 10xAI
    Apr 22 at 7:40
1
$\begingroup$

The shapes of your inputs/labels have been set up incorrectly which I am guessing is resulting in something funky happening in the calculations of loss.

You are setting input_shape=(None, None, 1) but your x_train has an input shape of (20,1,1,1). Firstly, these should have the same dimensions, with each None in the input shape indicating a variable batch dimension. The simple approach would be to have input_shape=(None, 1) and x_train have an input shape of (20,1) but alternatively having an input_shape=(None, None, None, 1) and x_train having an input shape of (20, 1, 1, 1) would also work. However, you must also ensure that y_train has the same batch dimensions as x_train. So in the simple case this would be (20, 1), but if x_train has a shape (20, 1, 1, 1) then y_train should also have a shape of (20, 1, 1, 1).

Making these changes I get the same val_loss during training as what is calculated in evaluation. The training loss (as opposed to val_loss) will always be different because training loss is calculated during training (prior to the weight updates).

$\endgroup$
8
  • $\begingroup$ The input_shape=(None, None, 1) is not an issue. As I had commented next to it, input_shape=(1, 1, 1) gives identical results. And the issue is not what happens with the model.fit validation loss values (they are never far from the model.evaluate values, but what happens with the model.fit training loss values. I have 1 batch of 2 examples for training. Every time this is run the optimizer drives the weights and biases to values that exactly match 1 example and then the training loss becomes 0, stopping further learning. It is like one of the two training examples is ignored. $\endgroup$ Apr 22 at 8:29
  • $\begingroup$ I tried all suggested changes, including the suggestion of a 4-dimensional output tensor, and the results don't change. Removing the reshaping into (20,1,1,1) and (20,1) tensors and changing the input_shape to (1), has the same results. I understand that "The training loss (as opposed to val_loss) will always be different because training loss is calculated during training (prior to the weight updates)." However, does this explain to have consistently the model.fit training loss driven to zero (false result) while the model.evaluate training loss stabilizes at the correct value of 0.054? $\endgroup$ Apr 22 at 8:48
  • $\begingroup$ Just to be clear (this may be what you did) - set the input_shape=(None, 1), and reshape BOTH x_train and y_train to (20, 1). Setting batch_size=18 (this is one training batch per epoch if your val set is 2 samples and total set is 20) and epochs=100 I get the following results: on the last training epoch training loss=0.0253 val_loss=0.0078 and the evaluation loss=0.02502, val loss=0.007781. $\endgroup$
    – James
    Apr 22 at 9:54
  • $\begingroup$ Great that you sorted out the issue, but just to be clear, this is not quite right. "Keras first takes the batch and then splits". No it does not. In your coding example you have set n_val=2, which means 2 of your 20 samples are validation samples and the 18 remaining are training samples. Setting your batch size to 18 will train for 1 iteration per epoch. Setting it to a batch size of 17 will train 2 iterations per epoch. A batch size of 2 leads to 9 iterations. Setting your batch size to 20 is larger than the maximum of 18 so it becomes 18 by default and trains for 1 iteration per epoch. $\endgroup$
    – James
    Apr 22 at 16:36
  • $\begingroup$ The batch size parameter of fit does refer only to the data that is used for training. If you check the validation_split documentation on keras it explains the behaviour of the validation split (keras.rstudio.com/reference/fit.html.) "The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on this data at the end of each epoch." The validation data is not taken per batch. So this comment is correct - "It first splits off the validation data and then takes the batch for training" - It does! $\endgroup$
    – James
    Apr 22 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.