I've trained 8 models based on the same architecture (convolutional neural network), and each uses a different data augmentation method. The accuracy of the models fluctuates greatly while the loss doesn't fluctuate as much. In one case, there was a 10% difference in accuracy while the loss value was exactly the same. Why would that be? Shouldn't they both change?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You might be winding up on the other side of the classification threshold. Remember that the neural network returns probabilities, not categories. If your predicted probability moves from $0.51$ in one model to $0.49$ in the other, that is a small change, but if you set the threshold for classification at $0.5$, the models give different categories. If you change from the right category to the wrong category, your accuracy score takes a big hit, yet the loss function is not affected much. I will demonstrate with the highly common binary crossentropy loss.
$$L(y_{\text{true}}, y_{\text{predicted}}=-y_{\text{true}}\log(y_{\text{predicted}})-(1-y_{\text{true}})\log(1-y_{\text{predicted}})\\ L(1, 0.51)=-1\log(0.51)-0\log(0.49)=0.673\\ L(1, 0.49)=-1\log(0.49)-0\log(0.51)=0.713 $$
This changes the loss by just $0.04$, yet the number of misclassifications changes by $1$!
This could be considered a drawback of threshold-based performance metrics like accuracy. I will include my usual links about proper scoring rules. (“Proper” is a technical term, not my judgment.)
https://stats.stackexchange.com/questions/357466/are-unbalanced-datasets-problematic-and-how-does-oversampling-purport-to-he https://www.fharrell.com/post/class-damage/ https://www.fharrell.com/post/classification/ https://stats.stackexchange.com/a/359936/247274 https://stats.stackexchange.com/questions/464636/proper-scoring-rule-when-there-is-a-decision-to-make-e-g-spam-vs-ham-email https://twitter.com/f2harrell/status/1062424969366462473?lang=en https://stats.stackexchange.com/questions/368949/example-when-using-accuracy-as-an-outcome-measure-will-lead-to-a-wrong-conclusio
