In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers?
Gradient vanishing means drastic decrease of gradients when backpropagating through many layers. This problem is also known for recurrent neural networks as they are mathematically equivalent to very deep networks.
That is true that gradients decrease during training. However, that is not gradient vanishing. That is the sign that the network has been trained, i.e. that is what you normally expect in the end of network training.