# Loss function returns x whereas tensorflow shows validation loss as (x+0.0567)

I have a custom loss function. In order to experiement how the loss is calculated during valiation, I update the loss function as follows:

def custom_loss(y_true, y_pred):
return 0 * tf.reduce_mean(input_tensor=-tf.reduce_sum(input_tensor=y_true_1 * tf.math.log(y_pred_1), axis=-1)) + 1

• Multiply by 0, to make the data dependent loss value as always 0.
• Add a constant value like 1 to it, so that the loss function always returns a value of 1.

Now when I run model.evaluate on my data, tensorflow shows me the loss value as 1.0567.

To further experiment, I updated the constant value being returned by loss function from 1 to other constant values. Below you can see the table:

• constant 0 => loss 0.0567
• constant 1 => loss 1.0567
• constant 2 => loss 2.0567

I was expecting the final calculated loss to be same as the constant value I passed. But that's not the case. What is the reason for a constant difference of 0.0567?

I have verified it across three versions of tensorflow: 1.14.0, 1.15.0, and 2.1.0.

• I'm not used to tf but looking at how the official loss functions are implemented here, they seem to convert constants using ops.convert_to_tensor(0.5, dtype=...), and then multiply them using math_ops.multiply. That might affect the backpropagation graph. (specifically I was looking at the huber_loss implementation) – A Kareem May 5 at 15:58
• Thanks @AKareem for your inputs. I think that doesn't apply to my situation though.. In my case, I'm using a custom loss function, which should bypass all of those layers. And those parameters like 0.5 is applicable only for some specific loss functions. – Vishal May 5 at 18:04
• Oh sure I understand your point, I meant just for experimental purposes try using ops.convert_to_tensor(0, dtype=...) and math_ops.multiply. – A Kareem May 6 at 1:29
• On further investigation, figured it out. One of my FC layers had a l2 regulariation as follows: model.add(layers.Dense(1024, activation='relu', kernel_regularizer=regularizers.l2(0.1)). Naturally when we add an L2 regularization like this, a correspnoding term will automatically get added to the loss function, which is what that 0.567 seems to be reflecting! – Vishal May 6 at 9:46
• If I comment out the l2 regularization from the model, the loss value goes down to zero, as expected... – Vishal May 6 at 9:47

The loss value is not only calculated by the response from loss function, but also the various regularizers added in the intermediate layers.
model.add(layers.Dense(1024, activation='relu', kernel_regularizer=regularizers.l2(0.1))