During hyperparameters tuning we select a metric to measure performance of the model. Example of metrics : f1 score, precision, recall, AUC ...

In general, for the training of neural networks, back-propagation tries to optimize the weights of the model according to the value of the loss function.

Here comes the question: Why don't people use the loss function as a main performance metric for neural networks optimization?


If you look for example at the L2 loss function:

$$ \sum (y - \hat{y})² $$

It takes into account how much the predicated value differs from the actual value. Note that because of the squaring of errors, L2 is sensitive to outliers. So, as you can see it takes into account how far our predictions are.

And that is exactly the problem: Most of the time we are not interested in how far our predictions are from the ground truth.

Imagine a real world application where the model predicts an image label. The application won't give you n examples together with their probabilities. It will just output the image with the highest probability.

That means, in the case of a classification problem where we use a softmax together with cross-entropy-loss it is not important if the predicted probability for a class is 0.49 or 0.01. We only care about if our model got the class correct or wrong.

Also, accuracy is much more tangible and better to interpret. That's why accuracy is used much more often.

| improve this answer | |
  • $\begingroup$ So basically, following the logic in the example, if we wanted to improve the probablities, we can use the losses for optimization? $\endgroup$ – ChiPlusPlus Dec 19 '18 at 8:54
  • 1
    $\begingroup$ Loss is most often used to guarantee the correctness of the back-propagation. We should see a decreasing curve for the training loss and, until some certain point, also for the validation loss. Neural nets optimize the weights to improve class probabilities by using a loss function. Just to recap, loss just represents a more accurate accuracy of your model and it is how neural nets train it. Accuracy on the other hand better represents the accuracy of a real world application and is better interpretable. That's why you should use accuracy for comparing models. $\endgroup$ – oezguensi Dec 19 '18 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.