Weird distribution of neural network outputs

Question

I've faced an unusual behavior during training a neural network.

The problem is to predict if a sample of 1st class or 2nd class. (2-class classification). Classes are imbalanced (~ 5 / 95). I use weighted crossentropy.

What do I see is probabilities of two classes for each sample, and it's seems weird to me that no matter how long I wait (number of epochs), the probabilities are in range [0, 0.9) and there are no samples close to probability 1.0. I guess, the imbalance and weighted loss + ADAM with low starting learning rate could led to this, but I am not sure.

Has anyone faced the same?

Answer 1

It's not that weird. You probably have an activation function on output that only reach 0 or 1 at infinity. To reach that 0 or 1 you would need to have your last hidden layer to output +/- infinity. Which is difficult given the weight structure. The best it can do will be to output extremely large positive or negative values.

It may be a problem because during the calibration process (which is an optimisation problem) you will spend significant ressources and time in trying to reach +/- infinity. Notably this is one of the causes of the famous exploding gradient problem.

How to deal with that ? In Efficient Backpropagation (LeCun and others, 1998), the author propose to use an activation function that is able to reach desired outputs. It states that a good activation function needs to have the following properties :

1. $f(\pm1)=\pm1$

2. The second derivative is a maximum at $x=1$

3. The effective gain is close to 1

Your problem is that you don't have something similar to the first condition. To respect those conditions LeCun proposes to use the activation function: $1.7159*tanh(\frac{2}{3}x)$. you might not need to use it in all your layers. Now your problem would probably be to find a framework that allows for custom activation functions.