1
$\begingroup$

I've trained 8 models based on the same architecture (convolutional neural network), and each uses a different data augmentation method. The accuracy of the models fluctuates greatly while the loss doesn't fluctuate as much. In one case, there was a 10% difference in accuracy while the loss value was exactly the same. Why would that be? Shouldn't they both change?

$\endgroup$

1 Answer 1

0
$\begingroup$

You might be winding up on the other side of the classification threshold. Remember that the neural network returns probabilities, not categories. If your predicted probability moves from $0.51$ in one model to $0.49$ in the other, that is a small change, but if you set the threshold for classification at $0.5$, the models give different categories. If you change from the right category to the wrong category, your accuracy score takes a big hit, yet the loss function is not affected much. I will demonstrate with the highly common binary crossentropy loss.

$$L(y_{\text{true}}, y_{\text{predicted}}=-y_{\text{true}}\log(y_{\text{predicted}})-(1-y_{\text{true}})\log(1-y_{\text{predicted}})\\ L(1, 0.51)=-1\log(0.51)-0\log(0.49)=0.673\\ L(1, 0.49)=-1\log(0.49)-0\log(0.51)=0.713 $$

This changes the loss by just $0.04$, yet the number of misclassifications changes by $1$!

This could be considered a drawback of threshold-based performance metrics like accuracy. I will include my usual links about proper scoring rules. (“Proper” is a technical term, not my judgment.)

https://stats.stackexchange.com/questions/357466/are-unbalanced-datasets-problematic-and-how-does-oversampling-purport-to-he https://www.fharrell.com/post/class-damage/ https://www.fharrell.com/post/classification/ https://stats.stackexchange.com/a/359936/247274 https://stats.stackexchange.com/questions/464636/proper-scoring-rule-when-there-is-a-decision-to-make-e-g-spam-vs-ham-email https://twitter.com/f2harrell/status/1062424969366462473?lang=en https://stats.stackexchange.com/questions/368949/example-when-using-accuracy-as-an-outcome-measure-will-lead-to-a-wrong-conclusio

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.