# Loss values seem to fluctuate, yet the weights are correct

I'm taking my first steps with tensorflow (and in ML in general), and using this piece of code to train a very simple model that tries to find the underlying linear relation: f(x,y) = 4x +7y -2 (+ noise drawn from a uniform distribution from -1 to 1).

model = tf.keras.Sequential([
tf.keras.layers.Dense(output_size,
kernel_initializer=tf.random_uniform_initializer(minval=-0.1, maxval=0.1),
bias_initializer=tf.random_uniform_initializer(minval=-0.1, maxval=0.1))
])

custom_optimizer = tf.keras.optimizers.SGD(learning_rate=0.03)
model.compile(optimizer=custom_optimizer, loss='mean_squared_error')
model.fit(training_data['inputs'], training_data['targets'], epochs=100, verbose=2)


I've tried several learning rates (lr) and found that for lr < 0.03, this loss in each iteration decreases to a certain point from which it remains constant, and for lr > 0.03 the loss is very high from the first iteration, reaching infinity soon after. for lr = 0.03, I get a loss values that seem to fluctuate, here it is for the last 10 iterations:

Epoch 90/100
32/32 - 0s - loss: 4.1204
Epoch 91/100
32/32 - 0s - loss: 1.3650
Epoch 92/100
32/32 - 0s - loss: 0.4946
Epoch 93/100
32/32 - 0s - loss: 1.3834
Epoch 94/100
32/32 - 0s - loss: 2.7705
Epoch 95/100
32/32 - 0s - loss: 2.3662
Epoch 96/100
32/32 - 0s - loss: 2.8347
Epoch 97/100
32/32 - 0s - loss: 0.7657
Epoch 98/100
32/32 - 0s - loss: 3.2244
Epoch 99/100
32/32 - 0s - loss: 2.9569
Epoch 100/100
32/32 - 0s - loss: 1.7180


The weights are very close to the actual weights (3.98 instead of 4, etc.) which is very similar to what I get when using lr < 0.03 (i.e., accuracy is quantitively the same).

I'm wondering how's that possible since the loss function doesn't seem to reach a certain limit as in lr < 0.03. Struggling to understand the underlying math that "causes" loss values to fluctuate yet still produce the correct weights.

Clarification: I know there's nothing "special" about lr = 0.03, but in my case this is the learning rate that caused this phenomenon.

• Can you share the different loss curves (loss plotted against epochs) for us to visualise them? For lr < 00.3, = 0.03 and > 0.03. Commented Nov 22, 2023 at 10:22
• I'm not sure this matches your case, it would be helpful to see the loss curves. But take a toy example where your function is just f(x) = 4x and your current weight is 3.98, after each update the gradient direction will flip. The resulting loss values might depend on whether you take loss to be the mean or the sum of the individual losses. Commented Nov 23, 2023 at 11:12
• If something similar is happening in your case, the oscillation is not constant because of the relationship between the two input variables (in the toy example there is only one input variable, hence constant oscillation). Commented Nov 23, 2023 at 11:21