1
$\begingroup$

The code below is a translation of Nielsen's first mnist code to Keras. Surprisingly, the last accuracy value of the .fit method and the accuracy value for the .evaluate method are different for the training data. As I expected, for the validation data the accuracies are the same. The same behavior is seen with losses. I would appreciate finding out what is going on.

import numpy as np
from tensorflow import keras
from tensorflow.keras import layers

num_classes = 10
input_shape = (28, 28, 1)

# the data, split between train and test sets
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
print(x_train.shape,y_train.shape,x_test.shape,y_test.shape)
# Scale images to the [0, 1] range
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255
# Make sure images have shape (28, 28, 1)
x_train = np.expand_dims(x_train, -1)  #adds another dimension at the end for color
x_test = np.expand_dims(x_test, -1)


# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

initializer = keras.initializers.RandomNormal(mean=0., stddev=1.)

model = keras.Sequential(
    [
        keras.Input(shape=input_shape),
        layers.Flatten(),
        layers.Dense(30, activation="sigmoid", use_bias=True,
                     kernel_initializer=initializer, 
                     bias_initializer=initializer),
        layers.Dense(num_classes, activation="sigmoid",
                     kernel_initializer=initializer, 
                     bias_initializer=initializer),    
    ]
)
batch_size = 10
epochs = 2
opt = keras.optimizers.SGD(learning_rate=3)
model.compile(loss="MeanSquaredError", optimizer=opt, metrics=["accuracy"])
ann=model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=1/6)

# Evaluate the trained model
print("\n")
score = model.evaluate(x_train[:50000], y_train[:50000], verbose=0)
print("evaluate train accuracy:", score[1])
score = model.evaluate(x_train[-10000:], y_train[-10000:], verbose=0)
print("evaluate validation accuracy:", score[1])
print("history last train accuracy:",ann.history['accuracy'][-1])
print("history last validation accuracy:",ann.history['val_accuracy'][-1])
$\endgroup$
1
  • 1
    $\begingroup$ Can you elaborate on what you refer to with “last accuracy”? $\endgroup$
    – hH1sG0n3
    Apr 11, 2021 at 8:54

1 Answer 1

2
$\begingroup$

training's loss/accuracy are not calculated for the whole dataset every time.
It is calculated for the current batch and averaged successively.
test's loss/accuracy is calculated at the end of an epoch for the whole test data.

Check these references
Keras FAQ
SO Answer

You may put the validation data same as the train data to check it.

ann=model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(x_train,y_train))

# Evaluate the trained model
print("\n")
score = model.evaluate(x_train, y_train, verbose=0)

enter image description here


Or, you may write a custom callback to print appropriate logs as suggested in the SO answer.

$\endgroup$
4
  • $\begingroup$ Many thanks 10xAI. I had not understood your answer. I incorrectly thought that Keras in model.fit first splits off the validation data and then takes the specified batch_size from the remaining training data. It first takes the specified batch-size and then splits off the validation data. In recall mode (model.evaluate) there is no splitting. This explains everything. My question has been satisfactorily answered and the thread should be closed with credit to 10xAI. $\endgroup$ Apr 22, 2021 at 16:03
  • $\begingroup$ That's great. Please accept this answer and will suggest you to close/delete the other question. $\endgroup$
    – 10xAI
    Apr 22, 2021 at 16:12
  • $\begingroup$ Unfortunately, I jumped the gun. Keras does first split off the validation data and then takes the specified batch_size from the remaining training data. $\endgroup$ Apr 22, 2021 at 18:47
  • $\begingroup$ But let's close this thread. I will simplify the code more to make the issue crystal clear $\endgroup$ Apr 22, 2021 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.