2
$\begingroup$

I think that the vanishing gradient problem occurs when the derivative of activation function is less than 1. The deeper the neural network is, the f' * f' * f'... operation will result in the gradient closer to zero.

However, if we use hard-sigmoid as an activation function (its derivative is 0 or 1), the vanishing gradient problem can be solved.

I think the dying problem (derivative = 0 @ ReLU) is different from the vanishing gradient problem.

Is it right?

$\endgroup$
2
$\begingroup$

The vanishing gradient problem appears with activation functions that squash their input in a very small output range, like sigmoids and tanh. The input range is large and therefore the probability for it to fall in the squashing part of the sigmoid is very high. Chain several sigmoids and there you have vanishing gradients,

Typical examples of such a type of activations are sigmoid and hyperbolic tangent. Hard sigmoids are no exception. Their main difference with normal sigmoids is that they are computationally cheaper.

ReLU's are different. Half of their range is linear. That's why they don't suffer from vanishing gradients. However, they have their own problem: dying ReLU. This happens when the input falls far in the left side and cannot escape.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ I think you mean that the derivative of activation function is less than 1 when activation functions squash their input in a small output range. In case of hard-sigmoid, the derivative can also be less than 1. However, if always 1 derivative is ensured, the vanishing gradient problem can be solved. How about your opinion? $\endgroup$ – S.Lim Sep 6 '17 at 2:31
  • $\begingroup$ What do you mean with "if always 1 derivative is ensured"? The derivative of hard sigmoid is 0 in the plateaus. $\endgroup$ – ncasas Sep 6 '17 at 7:04
  • $\begingroup$ I had a mistake. You are right. The derivative of hard-sigmoid is 0 in the plateaus. However, if we assume the derivative of hard-sigmoid is 1 in the linear region, the vanishing gradient problem can be solved in that region. The region where the derivative of hard-sigmoid is 0 can be considered as the dying ReLU. In other words, I think that the problem, which is occurred by the derivative which is less than 1, is not exist when using hard-sigmoid. $\endgroup$ – S.Lim Sep 6 '17 at 7:19
  • $\begingroup$ Yes, in the linear regime of the hard sigmoid there is no problem (just like ReLUs). I also agree that the situation with hard sigmoids is more similar to the dying ReLU problem. $\endgroup$ – ncasas Sep 6 '17 at 8:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.