In the case of a neural net with a relatively small training data set, doing simple classification with categorical cross entropy (log loss), it is very easy for the results of the network to be "overconfident" because that improves log loss on the training data.
What I mean by overconfident (this is partly intuition and partly something that can be derived from Bayesian stats): given 1000 training examples in 2 equal classes, it is unreasonable for any classifier to give a probability of a new input as being in one class or the other which is substantially higher than (1-1/1000), even if the classifier is perfect on the training data. That is because if the training set was increased by just one more example, the 1001st example could break whatever rule seems to have been followed in the data set up to that point, and that would force all the predicted values/probability estimates to be updated accordingly.
However, in practice trained neural nets with log loss function will output values which are arbitrarily close to 0 or 1 given a long enough training run, because that continues improving their performance on the training data way past the point that it makes sense to do so based on the size of the training data.
The question: how can the usual log loss function be modified so that making predictions extremely close to 0 or 1 does not hugely improve log loss compared to predictions which are only somewhat close?