7
$\begingroup$

As leaky relu does not lead any value to 0, so training always continues. And I can't think of any disadvantages it have.

Yet Leaky relu is less popular than Relu in real practice. Can someone tell why?

Improve this question
$\endgroup$
2
  • $\begingroup$ Where do you find that LeakyRelu is not so common in real practice/ $\endgroup$ – Carlos Mougan May 14 '20 at 8:27
  • 1
    $\begingroup$ @Carlos I have studied various models but never seen it being used $\endgroup$ – Prashant Gupta May 14 '20 at 11:31
2
$\begingroup$

Speaking from my experience, the performance of the two is almost the same. It might depend on the problem.

LeakyReLU was introduced to address the vanishing gradient problem, however it introduces yet another hyperparameter, the slope. If you want to squeeze out a little bit more performance of your model you can use LeakyReLU and tune the slope parameter, but that comes again of the cost of potential overfitting.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I thought that leaky relu uses a constant parameter for the negative values and the one that has one parameter more is the PRelu. At the end if a parameter is selected before the training and set to constant, is not a parameter more for the model $\endgroup$ – Carlos Mougan May 14 '20 at 8:25
  • 3
    $\begingroup$ You're right in some sense. PReLU is a general version of LeakyReLU (alpha=0.01) and ReLU (alpha=0). But alpha is still a hyperparameter which has to be selected before training like the learning rate for example. Although alpha can be treated as trainable parameter as well. ReLU just has computational advantages, is used for a longer time (better known) and the advantages of the other seem to be not large enough. I guess that's why ReLU is used most of the time. $\endgroup$ – Tinu May 14 '20 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.