Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Regarding Tensorflow/Keras SparseCategoricalcrossEntropy.
SparseCategoricalcrossEntropy(from_logits=True) expects the logits that has not been normalized by softmax. Then does SparseCategoricalcrossEntropy applies softmax internally?
I suppose applying softmax would not be required internally as taking argmax would give the index expected.
Please help clarify.
Yes, Softmax function is called when logit=True
Infact, if we check the keras code [Link], the softmax output is ignored in every condition and tf.nn.sparse_softmax_cross_entropy_with_logits is called. This function calculate softmax prior to cross_entropy as explained [Here]
I suppose applying softmax would not be required internally as taking argmax would give the index expected.
Predicting the class is not the only task here. It has to calculate the appropriate loss value to provide feedback to the network
If the target is in probability values[OHE] i.e. sum=1, not an open-ended logit i.e. [-inf, inf]. So the output should be similarly normalized prior to the loss calculation.
