I'm a bit confused by the coexistence of Loss and Accuracy metrics in Neural Networks. Both are supposed to render the "exactness" of the comparison of $y$ and $\hat{y}$, aren't they? So isn't the application of the two redundant in the training epochs? Moreover, why aren't they correlate?


Log loss has the nice property that it is a differentiable function. Accuracy might be more important and is definitely more interpretable but is not directly usable in the training of the network due to the backpropagation algorithm that requires the loss function to be differentiable. When your preferred loss is not directly optimizable (like the accuracy) you use a loss function that behaves similarly to proxy the true metric. In case of binary classification you would use a sigmoid at the end and a log loss to approximate accuracy. They are highly correlated.


Loss is more general than accuracy. In classification, you can go to 100% accuracy, where all the labels are predicted correctly. But what about regression or forecasting? There is no definition of 0% and 100%

Loss can be optimized with various methods. In Numerical Methods class, you've learned to solve a function by optimizing it (which is minimizing $|y_{hat}-y|$ ) with various methods such as Newton's method, bisection method, etc.


Yes, they both measure the exactness of y and y_hat and yes they're usually correlated. Sometimes the loss function might not be accuracy but you're still interested in measuring the accuracy even though you're not optimizing it directly. Google's TensorFlow MNIST example minimizes/optimizes cross entropy loss but displays accuracy to the user when reporting results, and this is completely fine.

Sometimes you don't want to optimize accuracy directly. For example, if you have serious class imbalance, your model will maximize accuracy by simply always picking the most common class, but this would not be a useful model. In this case entropy / log-loss would be a better loss function to optimize.

  • 7
    $\begingroup$ More importantly, accuracy is a not a differentiable function so you cannot backpropagate through it. $\endgroup$ Aug 25 '16 at 13:45
  • $\begingroup$ @JanvanderVegt Yes, that's a great point $\endgroup$
    – Ryan Zotti
    Aug 25 '16 at 13:45
  • $\begingroup$ I learnt that in Keras I can put a "custom" evaluation metrics (by custom in this case I mean that no built-in implementation in Keras, like AUC or F1-Score) in the compilation function. I presume that in this case this "custom" metrics will be used/displayed instead of Accuracy everywhere where show_accuracy parameter is set to True (like at fitting or in evaluation). Is that correct? $\endgroup$
    – Hendrik
    Aug 25 '16 at 14:10
  • 1
    $\begingroup$ @Hendrik yes you can, just create a def your_own_metric(y_true, y_pred) function and pass it to model.compile(..., metrics=[your_own_metric]) $\endgroup$ Aug 25 '16 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.