0
$\begingroup$

I'm not sure if this exists. Is there such a situation where weights in gradient descent fail to work or break up? If so, how and when?

$\endgroup$
4
  • $\begingroup$ Can you explain more your question? What do you mean by " weights in gradient descent fail to work"? $\endgroup$ Feb 14, 2019 at 9:51
  • $\begingroup$ for example, gradients of weights cease to exist? i dont know exactly. i was randomly asked about it $\endgroup$
    – feynman
    Feb 14, 2019 at 10:33
  • $\begingroup$ Do you mean vanishing or exploding gradients problem? $\endgroup$ Feb 14, 2019 at 10:34
  • $\begingroup$ yes, yes, probably $\endgroup$
    – feynman
    Feb 15, 2019 at 5:36

2 Answers 2

0
$\begingroup$

Yes, It is possible. I'll give an example. Consider we are using sigmoid activation for our network

enter image description here

The above given is the gradient descent update equation according to which new weights are obtained from old weights

CASE 1: When the product of partial derivatives becomes small

(Q) How will they the product become small??

enter image description here

See the derivative of sigmoid function, It doesn't go above 0.3, When multiple value less than or equal to 0.3 are multiplied, The overall product becomes so small. (Try multiplying 5 different values less than 0.3 in calculator). So you get a very small value and when this value is used to updated the weights in the topmost equation. You will notice that there is literally no update taking place to the weights. So weights are stuck there and hence model never converges or takes an eternity to converge....

This problem is called Vanishing gradients problems. This occurs if we use Sigmoid activation and similarly there is another problem called exploding gradients which occurs if we use Tan h activation. Hope its helps.

$\endgroup$
0
$\begingroup$

There is a situation called "exploding gradients" where very large error gradients accumulate and result in very large updates to neural network model weights during training.

This can result in an unstable model that cannot learn from the training data or even result in NaN weight values that can no longer be updated.

$\endgroup$
1
  • $\begingroup$ do u mean weight step lengths being too big as to surpass/overlook the loss function minimum? $\endgroup$
    – feynman
    Feb 15, 2019 at 5:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.