# What is the relationship between the accuracy and the loss in deep learning?

I have created three different models using deep learning for multi-class classification and each model gave me a different accuracy and loss value. The results of the testing model as the following:

• First Model: Accuracy: 98.1% Loss: 0.1882

• Second Model: Accuracy: 98.5% Loss: 0.0997

• Third Model: Accuracy: 99.1% Loss: 0.2544

My questions are:

• What is the relationship between the loss and accuracy values?

• Why the loss of the third model is the higher even though the accuracy is higher?

• Please familiarize yourself with proper scoring rules. Kolassa’s answer and links within here give a good rabbit role into which you can dive: stats.stackexchange.com/questions/464636/…. Briefly, accuracy is a highly problematic way of evaluating a classifier, counterintuitive as that seems.
– Dave
Sep 5, 2020 at 18:50

There is no relationship between these two metrics.
Loss can be seen as a distance between the true values of the problem and the values predicted by the model. Greater the loss is, more huge is the errors you made on the data.

Accuracy can be seen as the number of error you made on the data.

That means:

• a low accuracy and huge loss means you made huge errors on a lot of data
• a low accuracy but low loss means you made little errors on a lot of data
• a great accuracy with low loss means you made low errors on a few data (best case)
• your situation: a great accuracy but a huge loss, means you made huge errors on a few data.

For you case, the third model can correctly predict more examples, but on those where it was wrong, it made more errors (the distance between true value and predicted values is more huge).

NOTE:

Don't forget that low or huge loss is a subjective metric, which depends on the problem and the data. It's a distance between the true value of the prediction, and the prediction made by the model. It depends also on the loss you use.

Think:

• If your data are between 0 and 1, a loss of 0.5 is huge, but if your data are between 0 and 255, an error of 0.5 is low.
• Maybe think of cancer detection, and probability of detecting a cancer. Maybe an error of 0.1 is huge for this problem, whereas an error f 0.1 for image classification is fine.
• Thank you for your replying, if you mean by " data are between 0 and 1 or between 0 and 255 ", the scale !.. The image scale is 0-255. And the aim of the model is to classify images.
– N.IT
Dec 14, 2018 at 12:23
• Do you have a reference for these information ?
– N.IT
Dec 14, 2018 at 12:32
• @N.IT you did not specified in the question but my answer still stand as it is more general. Sorry I don't have reference for my claims. Dec 14, 2018 at 13:23
• "There is no relationship between these two metrics." isn't really accurate. Of course, there is a relationship between those two. Indeed, not a linear one. As @JérémyBlain noted, one can't really decide how well your model is based on the loss. That's why loss is mostly used to debug your training. Accuracy, better represents the real world application and is much more interpretable. But, you lose the information about the distances. A model with 2 classes that always predicts 0.51 for the true class would have the same accuracy as one that predicts 0.99. Dec 21, 2018 at 2:07
• @JérémyBlain. Thank you! I have huge loss as my data is between $[0,1]$ with 0.45. I am using mse on this data. What would you recommend please? I tied batch normalization, dropout layers, etc. but did not work.
– Avv
Dec 20, 2021 at 16:45

Actually, accuracy is a metric that can be applied to classification tasks only. It describes just what percentage of your test data are classified correctly. For example, you have binary classification cat or non-cats. If out of 100 test samples 95 is classified correctly (i.e. correctly determined if there's cat on the picture or not), then your accuracy is 95%. By the way, Confusion matrix describes your model much better then accuracy.

Loss depends on how you predict classes for your classification problem. For example, your model use probabilities to predict binary class cat or non-cats between 1 and 0. So if probability of cat is 0.6, then the probability of non-cat is 0.4. In this case, picture is classified as cat. Loss will be sum of the difference between predicted probability of the real class of the test picture and 1. In reality log loss is used for binary classification, I just gave the idea of what loss is.

• My model is a multi-class classification. Can I use loss as a measure?
– N.IT
Dec 14, 2018 at 12:49
• Sure, but loss doesn't show performance of your model in classification problem. What matters is whether your test sample is classified correctly or not. Loss matters to you only if you need probabilities of each class, instead of just class labels. Loss also can help you to improve your model by looking at huge loss cases. What matters more in multiclass classification if whether your classes are balanced, because usually models are biased towards larger classes. For example, f1-macro score shows how you model performs relatively to all classes. Dec 14, 2018 at 12:56

The other answers give good definitions of accuracy and loss. To answer your second question, consider this example:

We have a problem of classifying images from a balanced dataset as containing either cats or dogs. Classifier 1 gives the right answer in 80/100 of cases, whereas classifier 2 gets it right in 95/100. Here, classifier 2 obviously has the higher accuracy.

However, in the 80 of images classifier 1 gets right, it is extremely confident (for instance when it thinks an image is of a cat it is 100% sure that's the case), and in the 20 it gets wrong it was not at all confident (e.g. when it said a cat image contained a dog it was only 51% sure about that). In comparison, classifier 2 is extremely confident in its 5 wrong answers (it's 100% convinced that an image which actually shows a dog is a cat), and was not very confident about the 95 it got right. In this case, classifier 2 would have worse loss.

Someone says that accuracy has no relationship to the loss, but from a theoretical perspective, there IS a relationship.

Accuracy is $$1 - (error\ rate)$$ and the error rate can be seen as the expectation of the 0-1 loss: $$$$l_{01}(f(x), y) := \begin{cases} 0 & (f(x) = y) \\ 1 & (f(x) \neq y) \end{cases}$$$$

$$$$error\ rate = \mathbb{P}_{x, y} \left[ f(x) \neq y \right] = \mathbb{E}_{x, y} \left[ l_{01}(f(x), y) \right]$$$$ where $$f$$ is the model, $$x$$ is its input and $$y$$ is the ground truth label for $$x$$.

In order to maximize the accuracy, we want to minimize the error rate. However, due to the incontinuity of the 0-1 loss, it is practically impossible. Instead, a variety of "surrogate loss" is used. The surrogate loss function $$l$$ is required to have some properties:

• $$l$$ is continuous.
• $$l$$ is convex.
• $$l$$ bounds $$l_{01}$$ from above.

Surrogate losses with these properties allow us to minimize them via the well-known gradient descent algorithm.

Popular classes of those surrogate losses include the hinge loss that is used in support vector machine (SVM) and the logistic loss that is used in logistic regression and standard neural networks.

So, from a theoretical viewpoint, the accuracy and the loss displayed in every epoch of your training have some relationship. That is,

• Accuracy has a direct connection with the error rate, which we want to minimize in the training.
• Loss (usually the cross entropy loss, which is equivalent to the logistic loss in a sense) is a surrogate loss that bounds the error rate.