1. The cross entropy loss can be written as $L_1 = -\sum_i\sum_c y_{ic}\log P_{ic}$, where $i$ represents images and $c$ are the classes. $y_{ic}=1$ for the correct class.
  2. Instead of L_1 I can minimize the following $L_2 = -\sum_i\sum_c y_{ic}P_{ic}$

When I use these as the loss functions to CNN I found that for a particular problem $L_2$ performs significantly better than $L_1$. However, I am unable to explain it. What could be the reason for this?

  • 1
    $\begingroup$ Could you please confirm that the loss function in 2 is what you are referring to as L2 $\endgroup$ – mahesh ghanta Jun 19 '19 at 4:04
  • $\begingroup$ Also please share distribution of 0s and 1s in your dataset if that's ok $\endgroup$ – mahesh ghanta Jun 19 '19 at 4:06
  • $\begingroup$ I am using a weighted loss. I have 6 classes. So we can assume that all the classes contains almost equal number of images. $\endgroup$ – user570593 Jun 19 '19 at 5:50
  • 1
    $\begingroup$ In second case we are just making the value of negative as much as possible and solving it. The only reason I can think of is that we are penalizing the error with the true P value in L2 which is between 0 and 1 in this case vs 0 and infinity in the case of L1. My assumption is if any class is under represented L1 cannot wrongly predict that class still but L2 can ignore that class. $\endgroup$ – mahesh ghanta Jun 19 '19 at 8:13
  • 1
    $\begingroup$ It's an interesting problem. Please keep me posted if you find a reason. $\endgroup$ – mahesh ghanta Jun 19 '19 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.