I am dealing with numerical overflows and underflows with softmax and cross entropy function for multi-class classification using neural networks. Given logits, we can subtract the maximum logit for dealing with overflow but if the values of the logits are quite apart then one logit is going to be zero and others large negative numbers resulting in 100% probability for a single class and 0% for others. When loss is calculated as cross-entropy then if our NN predicts 0% probability for that class then the loss is NaN ($\infty$) which is correct theoretically since the surprise and the adjustment needed to make the network adapt is theoretically infinite.

I know this can be dealt with normalizing the data and choosing weights and biases from standard normal distribution but this is a what-if scenario where the data is mean preprocessed but not standard deviation processed, I believe this can also occur even after preprocessing both mean and stddev.

  • $\begingroup$ Many deep learning package has special functions to avoid such numerical instability caused by under/overflow. For example in Tensorflow it is suggested to use softmax_cross_entropy_with_logits instead of using softmax then calculating cross-entropy. $\endgroup$ – user12075 Sep 17 '18 at 22:35

Where exactly in the computations are these underflows manifesting? See here for a brief explanation around the extremes of the softmax.

Quick fixes could be to either increase the precision of your model (using 64-bit floats instead of, presumably, 32 bit floats), or just introduce a function that caps your values, so anything below zero or exactly zero is just made to be close enough to zero that the computer doesn't freak out. For example, use X = np.log(np.max(x, 1e-9)) before going into the softmax.

In any case, the softmax shouldn't have problems with your input, as the final activations are exponentiated:

$$ \sigma (\mathbf {z} )_{j}={\frac {e^{z_{j}}}{\sum _{k=1}^{K}e^{z_{k}}}} $$

This means all values will now be in the range [0, 1].

The cross-entropy equation should also be able to deal with the output of this.

If none of this helps your specific issue, could you share a specific example of your problem?

| improve this answer | |
  • 2
    $\begingroup$ My problem is after the softmax produces a result like [0, 0, 0, 1], the cross entropy just get a -inf because of the log $\endgroup$ – Eduardo Reis Apr 4 '19 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.