As per my knowledge, in back propagation- loss function or gradient is used to update the weights. in back propagation, weights became small w.r.t gradients, this leads to vanishing gradient problem.

can you please give insights about these two terms (gradient(SGD), exploding gradient problem, vanishing gradient problem).

how to select which activation function is useful/suitable at different layers?


1 Answer 1


To give you simple yet good answer:

  • Vanishing gradients is when the calculated gradients (derivatives) are so small that they become zero and
  • Exploding gradient is when the calculated gradients (derivatives) are so huge that it becomes infinity.

Now, how to solve these problems using different activation functions:

  • You will need to do some research on which activation to use but to give you a head start:
  • Sigmoid function and Tanh has these problems. But instead of them, you can use Relu or Leaky Relu. I am adding my notes that you can refer to understand more about the activation fuctions. And I hope I answered your question.

LINK: https://www.notion.so/Some-more-loss-functions-and-cost-functions-and-Activation-functions-6447509a554244bb857c82de28beb1af?pvs=4#88270c62aef74a36a97a04f6a40a5a9a

  • $\begingroup$ The link is private/protected and even after I login to google it is not authorized. Please test your links and also provide at least a summary in the question. I'll check back a bit later. $\endgroup$ Jun 24 at 18:48
  • $\begingroup$ try now. I changed it to public $\endgroup$ Jun 24 at 18:54
  • $\begingroup$ no dice, "you do not have access to this page. You're currently logged in as [my-google-email-address]" $\endgroup$ Jun 24 at 18:56
  • $\begingroup$ try this: notion.so/… $\endgroup$ Jun 24 at 19:07
  • $\begingroup$ same result - maybe get a friend to test it for you $\endgroup$ Jun 24 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.