Vanishing Gradient Problem:
- is during backpropagation, a gradient gets smaller and smaller or gets zero, multiplying small gradients together many times as going from output layer to input layer, then a model cannot be trained effectively.
- more easily occurs with more layers in a model.
- is easily caused by Sigmoid activation function because it produces the small values whose ranges are
0<=x<=1, then they are multiplied many times, making a gradient smaller and smaller as going from output layer to input layer. - can be detected if:
- parameters significantly change at the layers near output layer whereas parameters slightly change or stay unchanged at the layers near input layer.
- The weights of the layers near input layer are close to 0 or become 0.
- convergence is slow or stopped.
- can be mitigated using:
- Batch Normalization layer.
- Leaky ReLU activation function. *You can also use ReLU activation function but it sometimes cause Dying ReLU Problem which I explain later.
- PReLU activation function.
- ELU activation function.
- Gradient Clipping. *Gradient Clipping is the method to keep a gradient in a specified range.
- occurs in:
- CNN(Convolutional Neural Network).
- RNN(Recurrent Neural Network).
- doesn't easily occur in:
- LSTM(Long Short-Term Memory).
- GRU(Gated Recurrent Unit).
- Resnet(Residual Neural Network).
- Transformer.
- etc.
Exploding Gradients Problem:
- is during backpropagation, a gradient gets bigger and bigger, multiplying bigger gradients together many times as going from output layer to input layer, then convergence gets impossible.
- more easily occurs with more layers in a model.
- can be detected if:
- The weights of a model significantly increase.
- The weights of a model significantly increasing finally become NaN.
- convergence is fluctuating without finished.
- can be mitigated using:
- Batch Normalization layer.
- Gradient Clipping.
- occurs in:
- CNN.
- RNN.
- LSTM.
- GRU.
- doesn't easily occur in:
- Resnet.
- Transformer.
- etc.