0
$\begingroup$

I have a custom loss function. In order to experiement how the loss is calculated during valiation, I update the loss function as follows:

def custom_loss(y_true, y_pred):
    return 0 * tf.reduce_mean(input_tensor=-tf.reduce_sum(input_tensor=y_true_1 * tf.math.log(y_pred_1), axis=-1)) + 1 
  • Multiply by 0, to make the data dependent loss value as always 0.
  • Add a constant value like 1 to it, so that the loss function always returns a value of 1.

Now when I run model.evaluate on my data, tensorflow shows me the loss value as 1.0567.

To further experiment, I updated the constant value being returned by loss function from 1 to other constant values. Below you can see the table:

  • constant 0 => loss 0.0567
  • constant 1 => loss 1.0567
  • constant 2 => loss 2.0567

I was expecting the final calculated loss to be same as the constant value I passed. But that's not the case. What is the reason for a constant difference of 0.0567?

I have verified it across three versions of tensorflow: 1.14.0, 1.15.0, and 2.1.0.

$\endgroup$
  • $\begingroup$ I'm not used to tf but looking at how the official loss functions are implemented here, they seem to convert constants using ops.convert_to_tensor(0.5, dtype=...), and then multiply them using math_ops.multiply. That might affect the backpropagation graph. (specifically I was looking at the huber_loss implementation) $\endgroup$ – A Kareem May 5 at 15:58
  • $\begingroup$ Thanks @AKareem for your inputs. I think that doesn't apply to my situation though.. In my case, I'm using a custom loss function, which should bypass all of those layers. And those parameters like 0.5 is applicable only for some specific loss functions. $\endgroup$ – Vishal May 5 at 18:04
  • $\begingroup$ Oh sure I understand your point, I meant just for experimental purposes try using ops.convert_to_tensor(0, dtype=...) and math_ops.multiply. $\endgroup$ – A Kareem May 6 at 1:29
  • $\begingroup$ On further investigation, figured it out. One of my FC layers had a l2 regulariation as follows: model.add(layers.Dense(1024, activation='relu', kernel_regularizer=regularizers.l2(0.1)). Naturally when we add an L2 regularization like this, a correspnoding term will automatically get added to the loss function, which is what that 0.567 seems to be reflecting! $\endgroup$ – Vishal May 6 at 9:46
  • $\begingroup$ If I comment out the l2 regularization from the model, the loss value goes down to zero, as expected... $\endgroup$ – Vishal May 6 at 9:47
0
$\begingroup$

On further investigation was able to narrow it down.

The loss value is not only calculated by the response from loss function, but also the various regularizers added in the intermediate layers.

And one of the FC layers in my model was indeed having a L2 regularization term as below:

model.add(layers.Dense(1024, activation='relu', kernel_regularizer=regularizers.l2(0.1))

If I remove the l2 regularization, the loss goes indeed goes to zero if the loss function returns a zero.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.