As per my knowledge, in back propagation- loss function or gradient is used to update the weights. in back propagation, weights became small w.r.t gradients, this leads to vanishing gradient problem.

can you please give insights about these two terms (gradient(SGD), exploding gradient problem, vanishing gradient problem).

how to select which activation function is useful/suitable at different layers?


1 Answer 1


To give you simple yet good answer:

  • Vanishing gradients is when the calculated gradients (derivatives) are so small that they become zero and
  • Exploding gradient is when the calculated gradients (derivatives) are so huge that it becomes infinity.

Now, how to solve these problems using different activation functions:

  • You will need to do some research on which activation to use but to give you a head start:
  • Sigmoid function and Tanh has these problems. But instead of them, you can use Relu or Leaky Relu. I am adding my notes that you can refer to understand more about the activation fuctions. And I hope I answered your question.

LINK: https://www.notion.so/Some-more-loss-functions-and-cost-functions-and-Activation-functions-6447509a554244bb857c82de28beb1af?pvs=4#88270c62aef74a36a97a04f6a40a5a9a

  • $\begingroup$ The link is private/protected and even after I login to google it is not authorized. Please test your links and also provide at least a summary in the question. I'll check back a bit later. $\endgroup$ Commented Jun 24, 2023 at 18:48
  • $\begingroup$ try now. I changed it to public $\endgroup$ Commented Jun 24, 2023 at 18:54
  • $\begingroup$ no dice, "you do not have access to this page. You're currently logged in as [my-google-email-address]" $\endgroup$ Commented Jun 24, 2023 at 18:56
  • $\begingroup$ try this: notion.so/… $\endgroup$ Commented Jun 24, 2023 at 19:07
  • $\begingroup$ same result - maybe get a friend to test it for you $\endgroup$ Commented Jun 24, 2023 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.