My earlier question has unfortunately not been answered satisfactorily (as shown below if an epoch consists of a single batch the same strange behavior is observed), and this issue is driving me crazy, so please forgive the reposting of the question with new, more targeted, very simple code. Here is the link to my previous version of essentially the same question:
(I tried to comment to the proposed answer to that topic, but the allowed space is too short to enable posting of the new code).
Note that in the new example below an epoch has a single batch (2 examples) and the code is trained for a single epoch, so 1 epoch = 1 batch. Therefore what model.fit prints should match what model.evaluate prints. (This has been motivated by Andrej Karpathy's recommendation to start with fitting a single batch)
import numpy as np np.random.seed(42) import keras from keras import layers from keras.regularizers import l2 import tensorflow as tf tf.random.set_seed(42) import keras.backend as K #def mse_custom(y_true, y_pred): #Tried defining the loss, got same results # custom = K.mean(K.square((y_pred - y_true))) # return custom #Below I use 20 examples because the issue becomes much clearer than with a few examples x_train = np.array([0.10526316, 0. ,0.26315789,0.73684211,0.94736842,1., \ 0.52631579,0.84210526,0.21052632, 0.63157895,0.42105263,0.31578947,\ 0.36842105,0.78947368,0.15789474,0.68421053, 0.47368421,0.05263158,\ 0.57894737,0.89473684]).reshape(20,1,1,1) y_train = np.array([0.31438135, 0. ,0.48843161,0.83670357,0.86771850,1., \ 0.70921682,0.85732503,0.48475008,0.79387635,0.66044467,0.46540228,\ 0.57612215,0.97176724,0.43534982,0.83881293,0.61425264,0.30733072,\ 0.68508229,0.87707974]).reshape(20,1) input_shape = (None,None,1) #(1,1,1) gives the same results seedbase = 1 model = keras.Sequential( [ keras.Input(shape=input_shape), layers.Dense(100, activation="sigmoid", use_bias=True, kernel_initializer=keras.initializers.RandomNormal(mean=0., stddev=1./np.sqrt(1),seed=seedbase+0), #/np.sqrt(30) bias_initializer=keras.initializers.RandomNormal(mean=0., stddev=1.,seed=seedbase+1), kernel_regularizer=l2(0), bias_regularizer=None), layers.Dense(1, activation="sigmoid", use_bias=True, kernel_initializer=keras.initializers.RandomNormal(mean=0., stddev=1./np.sqrt(1),seed=seedbase+12), #/np.sqrt(10) bias_initializer=keras.initializers.RandomNormal(mean=0., stddev=1.,seed=seedbase+13), kernel_regularizer=l2(0), bias_regularizer=None), ] ) model.summary() """ ## Train the model """ n_val = 2 n_tr =20-n_val val_split = n_val/20 batch_size = 2 epochs = 1 opt = keras.optimizers.SGD(learning_rate=0.1) #opt='adam' #opt = keras.optimizers.RMSprop(0.00099) model.compile(loss='mean_squared_error', optimizer=opt) ann=model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=val_split, verbose = 1) """ ## Evaluate the trained model """ print("\n") score = model.evaluate(x_train[:n_tr], y_train[:n_tr], batch_size=1, verbose=0) print("Training loss:", score) score = model.evaluate(x_train[-n_val:], y_train[-n_val:], batch_size=1, verbose=0) print("Validation loss:", score)
And if you increase the epochs to say 100, the model.fit training loss goes to 0 (exactly matching only one of the two examples but very far from the other example, so clearly incorrect, and this is a clue to what is wrong) and training stops. The model.evaluate training loss stabilizes at the correct value. And the validation model.fit loss is very close to the validation model.evaluate loss.
Also, if I increase the number of examples in a single batch, the mismatch becomes much smaller.
I know I must be doing something wrong, but what? Your help would be very much appreciated.