Background: While fitting neural networks with relu activation, I found that sometimes the prediction becomes near constant. I believe that this is due to the relu neurons dieing during training as stated here. (What is the "dying ReLU" problem in neural networks?)

Question: What im hoping to do is to implement a check in the code itself to check if the neurons are dead. After that, the code could refit the network if needed.

As such, what is a good citeria to check for dead neurons? Currently im thinking of checking for low variance in the prediction as a citeria.

If it helps, im using keras.


2 Answers 2


A dead ReLU pretty much just means that its argument value is negative such that the gradient stays at 0; no matter how you train it from that point on. You can simply have a look at the gradient during training to see whether a ReLU is dead or not.

In practice you may simply want to use leaky ReLUs, i.e. instead of f(x) = max(0,x) you set f(x) = x if x > 0 and f(x) = 0.01x if x <= 0. This way you always allow a small non-zero gradient and the unit should not get fully stuck in training anymore.


A dead neuron is a neuron that does not update during training, ie. 0 gradient.

Keras allows gradient extraction directly for a given row of data. (Another nice example)

Or you can extract the neuron weights and calculate the gradient yourself
(eg. for relu, negative argument to relu -> 0 gradient.)

Unfortunately, gradient is data point specific. Only if the gradient is 0 for every row of training data can you be sure that the neuron will not update for all minibatches during a training epoch.

Leaky relu can be a helpful strategy since there's no value for leaky relu where the gradient equals 0.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.