44
$\begingroup$

I have created three different models using deep learning for multi-class classification and each model gave me a different accuracy and loss value. The results of the testing model as the following:

  • First Model: Accuracy: 98.1% Loss: 0.1882

  • Second Model: Accuracy: 98.5% Loss: 0.0997

  • Third Model: Accuracy: 99.1% Loss: 0.2544

My questions are:

  • What is the relationship between the loss and accuracy values?

  • Why the loss of the third model is the higher even though the accuracy is higher?

$\endgroup$
3

6 Answers 6

56
$\begingroup$

There is no relationship between these two metrics.

  • Loss can be seen as a distance between the true values of the problem and the values predicted by the model. The larger the loss, the larger the errors you made on the data.
  • Accuracy can be seen as the count of mistakes/misclassifications you made on the data. The larger the accuracy, the fewer misclassifications you made on the data.

That means:

  • large loss and small accuracy means you made huge errors on a lot of data (worst case)
  • small loss and small accuracy means you made small errors on a lot of data
  • small loss with a large accuracy means you made small errors on a few data (best case)
  • large loss but a large accuracy means you made huge errors on a few data (your case; the third model)

For your case, the third model can correctly predict more examples (large accuracy), but on those where it was wrong, it made larger errors (large loss - the distance between true value and predicted values is greater).

NOTE:

Don't forget that loss is a subjective metric, which depends on the problem and the data. It's a distance between the true value of the prediction, and the prediction made by the model.

  • The significance of the loss value is relative to the data itself; if your data are between 0 and 1, a loss of 0.5 is huge, but if your data are between 0 and 255, an error of 0.5 is low.
  • The acceptability of a loss value depends on the problem; consider cancer detection, where an error of 0.1 is unacceptably huge for this problem, whereas an error f 0.1 for image classification is fine.
$\endgroup$
6
  • $\begingroup$ Thank you for your replying, if you mean by " data are between 0 and 1 or between 0 and 255 ", the scale !.. The image scale is 0-255. And the aim of the model is to classify images. $\endgroup$
    – N.IT
    Dec 14, 2018 at 12:23
  • $\begingroup$ Do you have a reference for these information ? $\endgroup$
    – N.IT
    Dec 14, 2018 at 12:32
  • $\begingroup$ @N.IT you did not specified in the question but my answer still stand as it is more general. Sorry I don't have reference for my claims. $\endgroup$ Dec 14, 2018 at 13:23
  • 9
    $\begingroup$ "There is no relationship between these two metrics." isn't really accurate. Of course, there is a relationship between those two. Indeed, not a linear one. As @JérémyBlain noted, one can't really decide how well your model is based on the loss. That's why loss is mostly used to debug your training. Accuracy, better represents the real world application and is much more interpretable. But, you lose the information about the distances. A model with 2 classes that always predicts 0.51 for the true class would have the same accuracy as one that predicts 0.99. $\endgroup$
    – oezguensi
    Dec 21, 2018 at 2:07
  • $\begingroup$ @JérémyBlain. Thank you! I have huge loss as my data is between $[0,1]$ with 0.45. I am using mse on this data. What would you recommend please? I tied batch normalization, dropout layers, etc. but did not work. $\endgroup$
    – Avv
    Dec 20, 2021 at 16:45
12
$\begingroup$

Actually, accuracy is a metric that can be applied to classification tasks only. It describes just what percentage of your test data are classified correctly. For example, you have binary classification cat or non-cats. If out of 100 test samples 95 is classified correctly (i.e. correctly determined if there's cat on the picture or not), then your accuracy is 95%. By the way, Confusion matrix describes your model much better then accuracy.

Loss depends on how you predict classes for your classification problem. For example, your model use probabilities to predict binary class cat or non-cats between 1 and 0. So if probability of cat is 0.6, then the probability of non-cat is 0.4. In this case, picture is classified as cat. Loss will be sum of the difference between predicted probability of the real class of the test picture and 1. In reality log loss is used for binary classification, I just gave the idea of what loss is.

$\endgroup$
2
  • 1
    $\begingroup$ My model is a multi-class classification. Can I use loss as a measure? $\endgroup$
    – N.IT
    Dec 14, 2018 at 12:49
  • 3
    $\begingroup$ Sure, but loss doesn't show performance of your model in classification problem. What matters is whether your test sample is classified correctly or not. Loss matters to you only if you need probabilities of each class, instead of just class labels. Loss also can help you to improve your model by looking at huge loss cases. What matters more in multiclass classification if whether your classes are balanced, because usually models are biased towards larger classes. For example, f1-macro score shows how you model performs relatively to all classes. $\endgroup$ Dec 14, 2018 at 12:56
9
$\begingroup$

The other answers give good definitions of accuracy and loss. To answer your second question, consider this example:

We have a problem of classifying images from a balanced dataset as containing either cats or dogs. Classifier 1 gives the right answer in 80/100 of cases, whereas classifier 2 gets it right in 95/100. Here, classifier 2 obviously has the higher accuracy.

However, in the 80 of images classifier 1 gets right, it is extremely confident (for instance when it thinks an image is of a cat it is 100% sure that's the case), and in the 20 it gets wrong it was not at all confident (e.g. when it said a cat image contained a dog it was only 51% sure about that). In comparison, classifier 2 is extremely confident in its 5 wrong answers (it's 100% convinced that an image which actually shows a dog is a cat), and was not very confident about the 95 it got right. In this case, classifier 2 would have worse loss.

$\endgroup$
0
7
$\begingroup$

Someone says that accuracy has no relationship to the loss, but from a theoretical perspective, there IS a relationship.

Accuracy is $1 - (error\ rate)$ and the error rate can be seen as the expectation of the 0-1 loss: \begin{equation} l_{01}(f(x), y) := \begin{cases} 0 & (f(x) = y) \\ 1 & (f(x) \neq y) \end{cases} \end{equation}

\begin{equation} error\ rate = \mathbb{P}_{x, y} \left[ f(x) \neq y \right] = \mathbb{E}_{x, y} \left[ l_{01}(f(x), y) \right] \end{equation} where $f$ is the model, $x$ is its input and $y$ is the ground truth label for $x$.

In order to maximize the accuracy, we want to minimize the error rate. However, due to the incontinuity of the 0-1 loss, it is practically impossible. Instead, a variety of "surrogate loss" is used. The surrogate loss function $l$ is required to have some properties:

  • $l$ is continuous.
  • $l$ is convex.
  • $l$ bounds $l_{01}$ from above.

Surrogate losses with these properties allow us to minimize them via the well-known gradient descent algorithm.

Popular classes of those surrogate losses include the hinge loss that is used in support vector machine (SVM) and the logistic loss that is used in logistic regression and standard neural networks.

So, from a theoretical viewpoint, the accuracy and the loss displayed in every epoch of your training have some relationship. That is,

  • Accuracy has a direct connection with the error rate, which we want to minimize in the training.
  • Loss (usually the cross entropy loss, which is equivalent to the logistic loss in a sense) is a surrogate loss that bounds the error rate.
$\endgroup$
0
$\begingroup$

Ayúdenme a aclarar esta duda por favor

Después de ver este comentario: ------si sus datos están entre 0 y 1, una pérdida de 0,5 es enorme, pero si sus datos están entre 0 y 255, un error de 0,5 es bajo. Tengo las siguientes dudas:

Yo hice un modelo con la "Y" categórica, son 4 categorías y las clasifique: "0","1","2" y "3", ¿esta bien como lo clasifique (empieza en 0 y no en 1? y ¿mis datos estarían entre 0 - 4 para evaluar el test loss y accuracy?

este fue mi resultado: Test Loss: [0.9032219648361206, 3.8990705013275146]

$\endgroup$
0
$\begingroup$

In deep learning, during the training process, you typically monitor both the training and validation accuracy and loss to assess the performance of your model. Here's a brief explanation of each:

Training Accuracy and Loss:

Training Accuracy: This metric measures the percentage of correctly classified samples in the training dataset. It indicates how well the model is performing on the data it is being trained on. Training Loss: This metric represents a measure of how well the model is performing on the training data. It quantifies the difference between the predicted values and the actual values in the training dataset. The goal during training is to minimize this loss function.

Validation Accuracy and Loss:

Validation Accuracy: This metric measures the percentage of correctly classified samples in a separate validation dataset that the model hasn't seen during training. It provides an estimate of how well the model is generalizing to new, unseen data. Validation Loss: Similar to training loss, validation loss measures how well the model is performing on the validation dataset. It helps to identify overfitting or underfitting. Like training loss, the goal is to minimize this loss function, but its value might increase if the model starts overfitting to the training data. During the training process, these metrics are often plotted over epochs (iterations over the entire dataset) to visualize the performance of the model. Ideally, you want to see both training and validation accuracy increasing and both training and validation loss decreasing. If the training accuracy continues to increase while the validation accuracy stagnates or decreases, it might indicate overfitting. Similarly, if the training loss decreases while the validation loss increases, it might also indicate overfitting.

Monitoring these metrics helps in tuning hyperparameters, selecting models, and understanding how well the model is learning from the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.