Given a classifier using softmax, is it possible that say, for data point a which our model has correctly classified, has higher loss compared to data point b which is misclassified by our model?
If yes, please bring an example. If not, please prove.
================================================================
Update: I forgot to mention consider cross-entropy loss.
Assuming cross-entropy loss (as you specified in the comment), it can happens. The reason is that the loss in cross-entropy is "activated" only for the probability of the ground truth (t_i is 1 for the ground truth and 0 otherwise). Imagine a classification task with five classes, with the two following probability situations:
(a) 0.4 0.5 0.04 0.03 0.03
(b) 0.3 0.2 0.2 0.15 0.15
Assume that the true label is the first class.
In situation (a) the predicted class is the second one, but the probability of the first class (which is the only one which matters) is 0.4. In situation (b) the predicted class is the first one (so the right one), but the probability of the first class is less than the situation (a). Since cross-entropy cares only of the probability of the true label, in the first case you will obtain a lower loss because the probability is higher, even if the predicted class is the wrong one.
This phenomenon is more probable as the number of classes grows. For few classes, this is highly improbable or even impossible (think to the binary classification case, where this can't happen).
Cheers
SoftMax is not a LOSS Function. If this is binary classification task - then you are probably using binary cross entropy as the objective / loss function.
So - if the correct label is 1,
and you predict 0.7, then the loss is: -1 * log(0.7) = 0.35
But if you had made a worse prediction - say 0.1, then the loss is: -1 * log(0.3) = 2.3 (higher of the two)
