2
$\begingroup$

I've faced an unusual behavior during training a neural network.

The problem is to predict if a sample of 1st class or 2nd class. (2-class classification). Classes are imbalanced (~ 5 / 95). I use weighted crossentropy.

What do I see is probabilities of two classes for each sample, and it's seems weird to me that no matter how long I wait (number of epochs), the probabilities are in range [0, 0.9) and there are no samples close to probability 1.0. I guess, the imbalance and weighted loss + ADAM with low starting learning rate could led to this, but I am not sure.

Has anyone faced the same?

$\endgroup$
1
  • $\begingroup$ UPD. Nothing weird, it was really about weights and learning rate. $\endgroup$
    – Alexey
    Aug 19 '19 at 9:20
0
$\begingroup$

It's not that weird. You probably have an activation function on output that only reach 0 or 1 at infinity. To reach that 0 or 1 you would need to have your last hidden layer to output +/- infinity. Which is difficult given the weight structure. The best it can do will be to output extremely large positive or negative values.

It may be a problem because during the calibration process (which is an optimisation problem) you will spend significant ressources and time in trying to reach +/- infinity. Notably this is one of the causes of the famous exploding gradient problem.

How to deal with that ? In Efficient Backpropagation (LeCun and others, 1998), the author propose to use an activation function that is able to reach desired outputs. It states that a good activation function needs to have the following properties :

  1. $f(\pm1)=\pm1$

  2. The second derivative is a maximum at $x=1$

  3. The effective gain is close to 1

Your problem is that you don't have something similar to the first condition. To respect those conditions LeCun proposes to use the activation function: $1.7159*tanh(\frac{2}{3}x)$. you might not need to use it in all your layers. Now your problem would probably be to find a framework that allows for custom activation functions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.