0
$\begingroup$

I am doing some testing with tensorflow, and I bumbed into a very weird behaviour. Here is my code

fashion_mnist = tf.keras.datasets.fashion_mnist
(train_images1, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
train_images = train_images1[:32] / 255.0
train_labels = train_labels[:32]
test_images = test_images / 255.0

batch_size = 32
epochs     = 1

train_data = tf.data.Dataset.from_tensor_slices((train_images, train_labels)).batch(batch_size)
    
adam = tf.keras.optimizers.Adam(lr=0.002, beta_1=0.9, beta_2=0.999)

input_layer   = tf.keras.layers.Input(shape=(28,28,))
flatter       = tf.keras.layers.Flatten()(input_layer)
dense1        = tf.keras.layers.Dense(128,
                                     kernel_regularizer=tf.keras.regularizers.l2(0.01), 
                                     activation='relu')(flatter)                               
dense2        = tf.keras.layers.Dense(64,
                                     kernel_regularizer=tf.keras.regularizers.l2(0.01), 
                                     activation='relu')(dense1)
output_layer = tf.keras.layers.Dense(10,
                                     activation='softmax',name='output')(dense2)
model_naive = tf.keras.models.Model(inputs=input_layer,outputs=output_layer)

model_naive.compile(optimizer=adam,
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
              metrics=['accuracy'])
model_naive.summary()
history = model_naive.fit(x=train_data, validation_data=train_data, epochs=epochs)
model_naive.evaluate(train_data)

I simply import fashon_mnist (only one batch) and pass it as both train and validation data, to make comparison. No dropout in the network... so I would expect to find same loss and metric but this is the output

1/1 [==============================] - 1s 815ms/step - loss: 5.4532 - accuracy: 0.0312 - val_loss: 5.0106 - val_accuracy: 0.3125

To be sure I even did a model.evaluate() and this is what I find

1/1 [==============================] - 0s 12ms/step - loss: 5.0106 - accuracy: 0.3125

exactly the same found during training.

So, provided the evaluation is correct...what are these numbers "loss: 5.4532 - accuracy: 0.0312" ? I am using only one batch, so I would expect no averages over batches are involved.

Please help me to understand, this is driving me crazy. Thank you!

EDIT: With only 1 batch keras weirdly seems to print loss and score before applying gradients. The same does not happen with more than one batch where probably some strange average is performed. Still not solved the issue, any comment is still very welcomed!

$\endgroup$
0

1 Answer 1

0
$\begingroup$

This is because the calculation of the loss and accuracy are done before the first weight update (i.e. the model with the initialized parameters). After the loss is calculated the first time the loss is used to backpropagate the error throughout the network and to update the parameters. The loss is not calculated again during training (since this would just be an added training cost). The adjusted parameters are then used to calculate the loss for the next batch (training or validation). It would therefore make more sense to compare the training statistics of the second batch with the validation statistics of the first batch.

$\endgroup$
4
  • $\begingroup$ Thanks! Yes, I did what you suggested and it is like that. However, if more than one batch is involved this is not true anymore. Is there some batch loss average with updated weights? This is still not clear to me. $\endgroup$
    – Dave
    Commented Feb 9, 2021 at 10:15
  • $\begingroup$ Indeed, if you use more batches the metrics shown are the values for the whole epoch (i.e. averaged over all batches). $\endgroup$
    – Oxbowerce
    Commented Feb 9, 2021 at 10:18
  • $\begingroup$ Right, but the average loss among batches still doesn't match with the loss computed over the whole dataset. This is why I am wondering if the average over the batches is computed including the weight updates after each batch is processed (i.e. weights are not fixed along the average over batches, while they are in the loss computed over the dataset). $\endgroup$
    – Dave
    Commented Feb 9, 2021 at 10:22
  • $\begingroup$ The weights are indeed not fixed along the average over the batches, the weights are updated after each batch and these updated weights are used to calculate the metrics for the batch after that. $\endgroup$
    – Oxbowerce
    Commented Feb 9, 2021 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.