Dealing with extreme values in softmax cross entropy?

Question

I am dealing with numerical overflows and underflows with softmax and cross entropy function for multi-class classification using neural networks. Given logits, we can subtract the maximum logit for dealing with overflow but if the values of the logits are quite apart then one logit is going to be zero and others large negative numbers resulting in 100% probability for a single class and 0% for others. When loss is calculated as cross-entropy then if our NN predicts 0% probability for that class then the loss is NaN ($\infty$) which is correct theoretically since the surprise and the adjustment needed to make the network adapt is theoretically infinite.

I know this can be dealt with normalizing the data and choosing weights and biases from standard normal distribution but this is a what-if scenario where the data is mean preprocessed but not standard deviation processed, I believe this can also occur even after preprocessing both mean and stddev.

Answer 1

Where exactly in the computations are these underflows manifesting? See here for a brief explanation around the extremes of the softmax.

Quick fixes could be to either increase the precision of your model (using 64-bit floats instead of, presumably, 32 bit floats), or just introduce a function that caps your values, so anything below zero or exactly zero is just made to be close enough to zero that the computer doesn't freak out. For example, use X = np.log(np.max(x, 1e-9)) before going into the softmax.

In any case, the softmax shouldn't have problems with your input, as the final activations are exponentiated:

$$ \sigma (\mathbf {z} )_{j}={\frac {e^{z_{j}}}{\sum _{k=1}^{K}e^{z_{k}}}} $$

This means all values will now be in the range [0, 1].

The cross-entropy equation should also be able to deal with the output of this.

If none of this helps your specific issue, could you share a specific example of your problem?