Doubt in gradient , vanishing gradient problem in Back propagation

Question

As per my knowledge, in back propagation- loss function or gradient is used to update the weights. in back propagation, weights became small w.r.t gradients, this leads to vanishing gradient problem.

can you please give insights about these two terms (gradient(SGD), exploding gradient problem, vanishing gradient problem).

how to select which activation function is useful/suitable at different layers?

Answer 1

To give you simple yet good answer:

• Vanishing gradients is when the calculated gradients (derivatives) are so small that they become zero and
• Exploding gradient is when the calculated gradients (derivatives) are so huge that it becomes infinity.

Now, how to solve these problems using different activation functions:

• You will need to do some research on which activation to use but to give you a head start:
• Sigmoid function and Tanh has these problems. But instead of them, you can use Relu or Leaky Relu. I am adding my notes that you can refer to understand more about the activation fuctions. And I hope I answered your question.

LINK: https://www.notion.so/Some-more-loss-functions-and-cost-functions-and-Activation-functions-6447509a554244bb857c82de28beb1af?pvs=4#88270c62aef74a36a97a04f6a40a5a9a