How to prevent vanishing gradient or exploding gradient?

Question

What's causing the vanishing gradient or exploding gradient, and what are the measures to be taken to prevent it?

Answer 1

Vanishing gradient and exploding gradient are two common effects associated to training deep neural networks and their impact is usually stronger the deeper the network.

As you know, two fundamental operations when training neural networks are Forward-propagation and Back-propagation. When we carry out Back-propagation, that is, moving backward in the Network to calculate gradients of the loss function with respect to the weights, gradient values tend to decrease or increase dramatically, the further we get back in the network. This happens in cases when we have activation functions like Sigmoid, or TanH, whose non-linear regions below 0 (i.e. x << -5) and over 0 (i.e. x >> 5) return gradient values in the saturation regions. This means (x << -5) that neurons in earlier layers will learn in a very slow pace as compared to those layers located later in the network (vanishing gradients problem). Exploding gradients are the other side of the coin, i.e. when activation functions are saturated (with x >> 5), gradient values tend to increase dramatically, making updates to weights unstable, thus not being able to converge.

Some possible techniques to try to prevent these problems are, in order of relevance:

• Use ReLu - like activation functions: ReLu activation functions keep linearity for regions where sigmoid and TanH are saturated, thus responding better to gradient vanishing / exploding. You can also use different types like Leaky-ReLu, Randomized ReLu, etc.

• Use Batch Normalization (BN): this is another solution you could use in order to make your network more robust against gradient vanishing / exploding, especially if you are using sigmoid or TanH as activation functions. Actually, BN gives you more flexibility during the selection of the activation function for your network. The obtained architecture gets more robust at training, given that it is less prone to diverging due to initialization values or from higher learning rates.

• Reduce learning rate: if you increase your learning rate without considering using a ReLu-like activation function and/or not using BN, your network can diverge during training much more easily. By reducing your learning rate you can reduce the chance of suffering vanishing / exploding gradients problem, but your network will take longer to learn. That is why the first two options are located first in the list.

• Change your architecture: If you are using Convolutional Neural Networks, for example, and you are suffering from vanishing / exploding gradients, it might make sense to move to a new architecture like ResNETs. In comparison to other networks, these structures connect different layers between each other, i.e. the so-called skip connections, acting as gradient highways, allowing the gradient to flow between the different layers unhindered.

• Use proper weight initialization: you could use, for example, Xavier initialization Xavier et al. to reduce the chance of suffering vanishing / exploding gradients. By itself this option does not guarantee you will resolve these issues, but it makes your network more robust when combined with other methods.

• Gradient clipping: this can be used when having exploding gradient problem. Firsthand, we select a threshold value, and in case the value returned by the function of a gradient is greater than this threshold, we set it to a different value. You can check more info here.

Answer 2

@juanba1984 response explains the reasons well. But what the comment says about exploding gradient

when activation functions are saturated (with x >> 5), gradient values tend to increase dramatically

is not true. Saturated activation functions does not cause increase in gradients. When activation functions like Sigmoid or tanH saturate, gradient is close to zero

Answer 3

Vanishing Gradient Problem:

• is during backpropagation, a gradient gets smaller and smaller or gets zero, multiplying small gradients together many times as going from output layer to input layer, then a model cannot be trained effectively.
• more easily occurs with more layers in a model.
• is easily caused by Sigmoid activation function because it produces the small values whose ranges are 0<=x<=1, then they are multiplied many times, making a gradient smaller and smaller as going from output layer to input layer.
• can be detected if:
  • parameters significantly change at the layers near output layer whereas parameters slightly change or stay unchanged at the layers near input layer.
  • The weights of the layers near input layer are close to 0 or become 0.
  • convergence is slow or stopped.
• can be mitigated using:
  • Batch Normalization layer.
  • Leaky ReLU activation function. *You can also use ReLU activation function but it sometimes cause Dying ReLU Problem.
  • PReLU activation function.
  • ELU activation function.
  • Gradient Clipping. *Gradient Clipping is the method to keep a gradient in a specified range.
• occurs in:
  • CNN(Convolutional Neural Network).
  • RNN(Recurrent Neural Network).
• doesn't easily occur in:
  • LSTM(Long Short-Term Memory).
  • GRU(Gated Recurrent Unit).
  • Resnet(Residual Neural Network).
  • Transformer.
  • etc.

Exploding Gradients Problem:

• is during backpropagation, a gradient gets bigger and bigger, multiplying bigger gradients together many times as going from output layer to input layer, then convergence gets impossible.
• more easily occurs with more layers in a model.
• can be detected if:
  • The weights of a model significantly increase.
  • The weights of a model significantly increasing finally become NaN.
  • convergence is fluctuating without finished.
• can be mitigated using:
  • Batch Normalization layer.
  • Gradient Clipping.
• occurs in:
  • CNN.
  • RNN.
  • LSTM.
  • GRU.
• doesn't easily occur in:
  • Resnet.
  • Transformer.
  • etc.

Answer 4

Adding some summary points to @juanba1984 and @butwhy:

Selecting proper network architecture, proper weight initialization, selection of proper activation function, batch normalisation and proper optimiser with a well-tuned learning rate according to your network architecture are the common techniques to handle both of the gradient problems. Though they alone can't solve the issues completely but make the network more robust to these issues as mentioned by @juanba1984. We have to augment these with approaches like:

1. Reducing model complexity by reducing layers for vanishing gradient problems since the root cause of vanishing gradients lies in the multiplication of a bunch of small gradients, intuitively, it makes sense to fix this issue by reducing the number of gradients, i.e., reducing the number of layers in our network.

2. Gradient clipping and weight regularization for exploding gradient problems.

Hope this becomes useful for you.