In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ What do you mean by "rather that only vanishing by different layers"? $\endgroup$– Ethan YunMar 11, 2021 at 19:49
-
$\begingroup$ Sorry, I mean "Vanishing through different layers" I expressed myself badly $\endgroup$– Domenico BagnatoMar 11, 2021 at 19:50
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Gradient vanishing means drastic decrease of gradients when backpropagating through many layers. This problem is also known for recurrent neural networks as they are mathematically equivalent to very deep networks.
That is true that gradients decrease during training. However, that is not gradient vanishing. That is the sign that the network has been trained, i.e. that is what you normally expect in the end of network training.
