43
$\begingroup$

In the documentation it has been mentioned that y_pred needs to be in the range of [-inf to inf] when from_logits=True. I truly didn't understand what this means, since the probabilities need to be in the range of 0 to 1! Can someone please explain in simple words the effect of using from_logits=True?

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])
$\endgroup$

2 Answers 2

61
$\begingroup$

The from_logits=True attribute inform the loss function that the output values generated by the model are not normalized, a.k.a. logits. In other words, the softmax function has not been applied on them to produce a probability distribution. Therefore, the output layer in this case does not have a softmax activation function:

out = tf.keras.layers.Dense(n_units)  # <-- linear activation function

The softmax function would be automatically applied on the output values by the loss function. Therefore, this does not make a difference with the scenario when you use from_logits=False (default) and a softmax activation function on last layer; however, in some cases, this might help with numerical stability during training of the model. You may also find this and this answers relevant and useful about the numerical stability when from_logits=True.

$\endgroup$
3
  • 16
    $\begingroup$ so basically what it means is if softmax layer is not being added at the last layer then we need to have the from_logits=True to indicate the probabilities are not normalized , $\endgroup$ Apr 29, 2020 at 20:05
  • 4
    $\begingroup$ @NagendraPrasad That's right. $\endgroup$
    – today
    Apr 30, 2020 at 3:11
  • $\begingroup$ perfect explanation. $\endgroup$ May 19, 2021 at 21:57
1
$\begingroup$

When specifying from_logits=True basically means that when calculating error on dataset using the cost function J, where it depends how you define your J where J also depends on some function of X ( dataset ), then instead of calculating f(x) and then putting it in J, we can directly put f(x) in J. Now this might seem a bit confusing. Lets see in detail. So basically Logit means a simple Logistic regression unit used in Neural Networks as a Neuron. Now we calculated error associated with each Neuron of a layer and so on for all layers. For a simple neuron/Logit, the cost function J is defined as J = 1/m Σ-ylog(f(x))-(1-y)log(1-f(x)) , where m is size or number of training examples, y is given predictions and f(x) is lets say sigmoid function. Now sigmoid function is defined as f(x) = 1/(1+e^(-x*w+b) where x is the input for that layer, w are the weights associated with that neuron and b is the bias associated with that neuron. Normally when from_logits=False, then first f(x) is calculated and then put in the formula for J but when from_logits = True, then f(x) is directly put into the formula J. Now it might seem that both are the same thing but this is actually not the case. The result could be same if you are lucky but due to precision errors in decimal points, the value of J could come out to be different. This can effect the accuracy of your model. Thus when you directly place formula for f(x) into J, then tensorflow actually rearranges the terms to perform better calculations. This also kills any error that might arise from precision errors. If to say in very simple words : when from_logits = False, then J = 1/m Σ-ylog(f(x))-(1-y)log(1-f(x)) when from_logits = True, then J = 1/m Σ-ylog(1/(1+e^(-x*w+b))-(1-y)log(1-1/(1+e^(-x*w+b))

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.