I am working on the classification problem where by I am having a hinge loss function + other loss terms to optimize for which the input is the output from tanh layer at the end. But I can't reveal all the terms of loss, just can mention that it has representation learning loss + hinge loss + regularization terms. It is a binary classification problem. The weird behaviour is that the training loss is decreasing with every epoch, but training accuracy is also decreasing in third or fourth place after decimal point. What could be the probable cause? I am unable to understand this behaviour and the test accuracy is little higher than training accuracy after the complete training. for reference, I am attaching the loss plot and accuracy curves in the same order.

training loss plot

training accuracy curve

what is the reason for decreasing training accuracy? Due to this I feel I am not getting good results.

  • $\begingroup$ Could you tell us a little bit more about the classification problem? Do you have many or only two classes? And what does the loss function look like exactly? $\endgroup$ Commented Jul 18, 2022 at 13:42
  • $\begingroup$ @NiklasvMoers : I have edited the post, pls see. $\endgroup$ Commented Jul 18, 2022 at 15:29
  • 1
    $\begingroup$ One possible problem might be that your loss function favors one of the two predictions and your model is overfitting. As an easy example, consider the loss function $f(x, \hat{x}) = 1_{\{x \neq \hat{x}\}} \cdot (w_A 1_{\{x = A\}} + w_B 1_{\{x=B\}})$ with two classes A and B and two different weights $w_A$ and $w_B$. If $w_A > w_B$, your loss function encourages guessing A because it punishes guessing B when the actual class is A more than it punishes guessing A when the actual class is B. $\endgroup$ Commented Jul 18, 2022 at 15:38
  • 2
    $\begingroup$ As you are dealing with a binary classification problem, the loss function can only take 4 different values with $f(x, x) = 0$, so you only need to calculate $f(A, B)$ and $f(B, A)$. If those two are not the same, the above scenario applies. $\endgroup$ Commented Jul 18, 2022 at 15:40


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.