1
$\begingroup$

I have just made an MNIST classifier from a tutorial. It has 1 layer, no hidden layers. It uses PyTorch's nn.CrossEntropyLoss as a loss function. Plotting my losses after training, it starts out at around 2.25 and goes down to 0.5. What does this number mean, in a colloquial sense? Does a loss of 0.5 mean that it's right 50% of the time?

$\endgroup$
2
$\begingroup$

The cross entropy is equivalent (up to a constant) to the Kullback Leibler Divergence which has an interpretation based on information theory: (very crudely) it is the amount of information "lost" by representing the true labels via the distribution of their predicted values (measured in "nats" (e) or "bits" (2) depending on the base of the logarithm). I don't find it very useful as a "colloquial" interpretation or to think in units of bits.

Typically, you would calculate more interpretable metrics like accuracy and inspect them along with the loss to get a picture of the performance.

However, you can use the loss and compare it to a simple baseline (e.g. the model only predicting the average of your labels or only the majority class) and thus get a relative performance estimate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.