As leaky relu does not lead any value to 0, so training always continues. And I can't think of any disadvantages it have.
Yet Leaky relu is less popular than Relu in real practice. Can someone tell why?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Where do you find that LeakyRelu is not so common in real practice/ $\endgroup$ – Carlos Mougan May 14 '20 at 8:27
-
1$\begingroup$ @Carlos I have studied various models but never seen it being used $\endgroup$ – Prashant Gupta May 14 '20 at 11:31
Add a comment
|
$\begingroup$
$\endgroup$
2
Speaking from my experience, the performance of the two is almost the same. It might depend on the problem.
LeakyReLU was introduced to address the vanishing gradient problem, however it introduces yet another hyperparameter, the slope. If you want to squeeze out a little bit more performance of your model you can use LeakyReLU and tune the slope parameter, but that comes again of the cost of potential overfitting.
-
$\begingroup$ I thought that leaky relu uses a constant parameter for the negative values and the one that has one parameter more is the PRelu. At the end if a parameter is selected before the training and set to constant, is not a parameter more for the model $\endgroup$ – Carlos Mougan May 14 '20 at 8:25
-
3$\begingroup$ You're right in some sense. PReLU is a general version of LeakyReLU (alpha=0.01) and ReLU (alpha=0). But alpha is still a hyperparameter which has to be selected before training like the learning rate for example. Although alpha can be treated as trainable parameter as well. ReLU just has computational advantages, is used for a longer time (better known) and the advantages of the other seem to be not large enough. I guess that's why ReLU is used most of the time. $\endgroup$ – Tinu May 14 '20 at 9:05
$\begingroup$
$\endgroup$
Leaky relu is a way to overcome the vanishing gradients buts as you increase the slope from 0 to 1 your activation function becomes linear, you can try to plot a leaky relu with different slopes in negative part. The problem is losing non-linearity with in cost of having a better gradient back propagation. If you can get a good result with relu, switching to leaky relu may result in getting worse.
