In the documentation it has been mentioned that y_pred needs to be in the range of [-inf to inf] when from_logits=True. I truly didn't understand what this means, since the probabilities need to be in the range of 0 to 1! Can someone please explain in simple words the effect of using from_logits=True?
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
A very detailed explanation about the difference between from_logits=False and from_logits=True is provided here: https://stackoverflow.com/a/71365020/1661267
The from_logits=True attribute inform the loss function that the output values generated by the model are not normalized, a.k.a. logits. In other words, the softmax function has not been applied on them to produce a probability distribution. Therefore, the output layer in this case does not have a softmax activation function:
out = tf.keras.layers.Dense(n_units) # <-- linear activation function
The softmax function would be automatically applied on the output values by the loss function. Therefore, this does not make a difference with the scenario when you use from_logits=False (default) and a softmax activation function on last layer; however, in some cases, this might help with numerical stability during training of the model. You may also find this and this answers relevant and useful about the numerical stability when from_logits=True.
from_normalized or input_is_normalized rather than the rather odd, confusing, off-point and far-from-universal sub-case from_logits.
$\endgroup$
Commented
May 10, 2023 at 14:45
When specifying from_logits=True basically means that when calculating error on dataset using the cost function J, where it depends how you define your J where J also depends on some function of X ( dataset ), then instead of calculating f(x) and then putting it in J, we can directly put f(x) in J. Now this might seem a bit confusing. Lets see in detail. So basically Logit means a simple Logistic regression unit used in Neural Networks as a Neuron. Now we calculated error associated with each Neuron of a layer and so on for all layers. For a simple neuron/Logit, the cost function J is defined as J = 1/m Σ-ylog(f(x))-(1-y)log(1-f(x)) , where m is size or number of training examples, y is given predictions and f(x) is lets say sigmoid function. Now sigmoid function is defined as f(x) = 1/(1+e^(-x*w+b) where x is the input for that layer, w are the weights associated with that neuron and b is the bias associated with that neuron. Normally when from_logits=False, then first f(x) is calculated and then put in the formula for J but when from_logits = True, then f(x) is directly put into the formula J. Now it might seem that both are the same thing but this is actually not the case. The result could be same if you are lucky but due to precision errors in decimal points, the value of J could come out to be different. This can effect the accuracy of your model. Thus when you directly place formula for f(x) into J, then tensorflow actually rearranges the terms to perform better calculations. This also kills any error that might arise from precision errors. If to say in very simple words :
when from_logits = False, then J = 1/m Σ-ylog(f(x))-(1-y)log(1-f(x))
when from_logits = True, then J = 1/m Σ-ylog(1/(1+e^(-x*w+b))-(1-y)log(1-1/(1+e^(-x*w+b))
