0
$\begingroup$

Relu tends to show better convergence performance on gradient descent optimization than sigmoid activation function. As far I came to know that when Z approaches less than 0 then updation with gradient descent becomes too slow, But relu has also gradient 0 when z is less than 0 then what is difference ?

$\endgroup$
3
  • 1
    $\begingroup$ The following post explains pretty well. $\endgroup$
    – ofir1080
    Aug 16, 2021 at 16:20
  • $\begingroup$ Can you share a source ? I am not convinced this i s a general truth that relu show better convergence than sigmoid. $\endgroup$ Aug 17, 2021 at 8:09
  • $\begingroup$ This is covered in the lecture of Andrew NG's deep learning courses. Week 2 of the first course Neural Network and Deep Learning. $\endgroup$ Aug 20, 2021 at 16:04

1 Answer 1

2
$\begingroup$

Sigmoid

enter image description here

$f(x) = \frac{1}{1+e^{-x}}$

$f'(x) = f(x)(1-f(x)) $

When the value of sigmoid function is either too high or too low, the derivative becomes too small(close to zero). While error is back propagating in sigmoid activated neural networks, gradient degradation happens and it results in vanishing gradient.

Relu

enter image description here

$f(x) = max(0,x) $
$f'(x) = (0~if~x <0;1~if~x>0 ) $ While error is back-propagating in relu activated neural networks, gradient is not getting degraded.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.