I am doing some testing with tensorflow, and I bumbed into a very weird behaviour. Here is my code
fashion_mnist = tf.keras.datasets.fashion_mnist
(train_images1, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
train_images = train_images1[:32] / 255.0
train_labels = train_labels[:32]
test_images = test_images / 255.0
batch_size = 32
epochs = 1
train_data = tf.data.Dataset.from_tensor_slices((train_images, train_labels)).batch(batch_size)
adam = tf.keras.optimizers.Adam(lr=0.002, beta_1=0.9, beta_2=0.999)
input_layer = tf.keras.layers.Input(shape=(28,28,))
flatter = tf.keras.layers.Flatten()(input_layer)
dense1 = tf.keras.layers.Dense(128,
kernel_regularizer=tf.keras.regularizers.l2(0.01),
activation='relu')(flatter)
dense2 = tf.keras.layers.Dense(64,
kernel_regularizer=tf.keras.regularizers.l2(0.01),
activation='relu')(dense1)
output_layer = tf.keras.layers.Dense(10,
activation='softmax',name='output')(dense2)
model_naive = tf.keras.models.Model(inputs=input_layer,outputs=output_layer)
model_naive.compile(optimizer=adam,
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
metrics=['accuracy'])
model_naive.summary()
history = model_naive.fit(x=train_data, validation_data=train_data, epochs=epochs)
model_naive.evaluate(train_data)
I simply import fashon_mnist (only one batch) and pass it as both train and validation data, to make comparison. No dropout in the network... so I would expect to find same loss and metric but this is the output
1/1 [==============================] - 1s 815ms/step - loss: 5.4532 - accuracy: 0.0312 - val_loss: 5.0106 - val_accuracy: 0.3125
To be sure I even did a model.evaluate() and this is what I find
1/1 [==============================] - 0s 12ms/step - loss: 5.0106 - accuracy: 0.3125
exactly the same found during training.
So, provided the evaluation is correct...what are these numbers "loss: 5.4532 - accuracy: 0.0312" ? I am using only one batch, so I would expect no averages over batches are involved.
Please help me to understand, this is driving me crazy. Thank you!
EDIT: With only 1 batch keras weirdly seems to print loss and score before applying gradients. The same does not happen with more than one batch where probably some strange average is performed. Still not solved the issue, any comment is still very welcomed!