I'm trying to figure out the essence of the concepts "vanishing gradient and exploding gradient problem" in terms of real-world input-output training examples instead of in terms of the properties of the choice of activation function.
Can anybody direct to a good tutorial that include such examples?
I always refer "Andrew NG" documents if available for any ML understanding. I believe that Youtube link Vanishing and Exploding Gradient, will help you in understanding the concept in a better way.
Still a brief from my side:
DNN of K layers without an activation function, will be like multiplying K coefficients(weights) together. Somehow if you received initial coefficient value < 0, in such a case during back propagation till you reach to the starting layers (from Input layer side), it may possible you completely lost the value of gradient, as gradually you are multiplying smaller value i.e. vanishing a value. Similarly if initial coefficient value > 0, multiplying together will form a very big number i.e. exploding a value.
A good way to understand and intuitively comprehend the concept of vanishing gradients and exploding gradient would be through manually solve through a backpropagation. Since, Feed Forward Neural Network is simplest of all and Mostly sigmoid function and Tanh suffers from vanishing gradient . It would be wise to build a MLP with at least one hidden layer and compute the change in parameter values after forward pass , error calculation and backward pass to update weights and biases initialized randomly. https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/
Likewise, RNN are mostly suffering from exploding gradient you could apply same method.
It may seems far fetched to go to this trouble for understanding concepts , but it is worth your time.
At high level, you can think of vanishing gradients in the way Chinese whispers work: Part of the original information is being lost every time it is being passed backwards to another person. In a similar way, RNN architecture "looses" part of the original information of a gradient as it is being propagated from the very last time step backwards to the very first step.
Drilling down to the specifics see below:
Traditional Recurrent Neural Networks (RNN) have the ability to model sequential events by propagating through time, i.e. forward and backward propagation. RNN models connect each time-step (e.g. position of a word in a sentence) using the following function defined as hidden state:
$a_n = f(W_n, a_{n-1}, x_n)$
The hidden state $a_n$ carries past information by applying a linear combination over the previous step and the current input.
The issue with the above is that the hidden state of every current position is a function of all previous positions. This means when you backpropagate gradients through time (see BPTT) the gradient inherently "looses" part of its "amplitude" because of the chain rule in $a_n$:
$a_n = f(W_n, a_{n-1}, x_n) = f(W_n, f(W_{n-1}, a_{n-2}, x_{n-1}), x_n)$, since $ a_{n-1}=f(W_n, a_{n-2}, x_n)$.
In this way, the longer the input sequence is the worse long terms dependencies will be captured to to the way gradients vanish because of the chain rule in their hidden state.
I hope it helps. See here my relevant post in case it may also be of help https://datascience.stackexchange.com/a/84409/102852
