I am trying to train a deep neural network but I am having dying ReLU problem. I am using leaky Relu but still have the same problem. Isn't leaky relu supposed to not have such problems?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Actually, it is not possible that leaky ReLU suffers from a dying problem unless the slope of the negative inputs is so small. Try to use something like 0.2 for the slope of the negative part.
On the other hand, I believe your problem is not dying ReLU, because if it happens, almost all of your activations will output zero. My answer is basically telling you that your outputs may be so small and they change so slowly.
answered Nov 26, 2020 at 15:33