I have a model with a few outputs, each output with shape:

  • Shape: (batch_size, labels_1) -> softmax -> categorical_crossentropy -> loss_value_1
  • Shape: (batch_size, labels_2) -> softmax -> categorical_crossentropy -> loss_value_2
  • Shape: (batch_size, labels_3) -> softmax -> categorical_crossentropy -> loss_value_3
  • ....

Now this model is used for a classification problem, so I'm applying a softmax activation to each and using the categorical_crossentropy (or just "crossentropy") loss.

Basically, everything is working reasonably well, but I noticed the losses are unbalanced. (Not because of class unbalance, but because of a different number of neurons)

An output with more neurons tend to have greater losses because their initial predictions for the correct label (the only label considered by this loss) is closer to zero.

I could try to balance them with a simple loss / number_of_neurons, but I'm not confident that this will indeed balance the losses (since softmax is not linear), it sounds like unbalancing in a different way.

Is there a mathematically correct way to balance these losses to compensate for their number of neurons?

  • $\begingroup$ First of all, which framework/package are you using? $\endgroup$ – Valentin Calomme Dec 4 '19 at 14:06
  • 1
    $\begingroup$ Not relevant. It's both tensorflow.keras and pytorch. This is a mathematical question. $\endgroup$ – Daniel Möller Dec 4 '19 at 14:27
  • 1
    $\begingroup$ Maybe you find this introduction to Cross Entropy useful with respect to multi-class classification in a CNN framework. $\endgroup$ – Maeaex1 Dec 4 '19 at 15:57
  • $\begingroup$ Out of curiosity, why do you want to balance your losses in the first place? $\endgroup$ – Valentin Calomme Dec 4 '19 at 16:26
  • 1
    $\begingroup$ The balance is to assure that one output doesn't get favored in training (if one output performs better than another, it should mean the model is better for that kind of output, not because there are loss weights acting) $\endgroup$ – Daniel Möller Dec 4 '19 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.