As Leaky ReLU does not lead any value to 0, so training always continues. And I can't think of any disadvantages it has.
Yet Leaky ReLU is less popular than ReLU in real practice. Can someone tell why this is?
$\begingroup$
$\endgroup$
0
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
Speaking from my experience, the performance of the two is almost the same. It might depend on the problem.
LeakyReLU was introduced to address the vanishing gradient problem, however it introduces yet another hyperparameter, the slope. If you want to squeeze out a little bit more performance of your model you can use LeakyReLU and tune the slope parameter, but that comes again of the cost of potential overfitting.
-
$\begingroup$ I thought that leaky relu uses a constant parameter for the negative values and the one that has one parameter more is the PRelu. At the end if a parameter is selected before the training and set to constant, is not a parameter more for the model $\endgroup$ Commented May 14, 2020 at 8:25
-
4$\begingroup$ You're right in some sense. PReLU is a general version of LeakyReLU (alpha=0.01) and ReLU (alpha=0). But alpha is still a hyperparameter which has to be selected before training like the learning rate for example. Although alpha can be treated as trainable parameter as well. ReLU just has computational advantages, is used for a longer time (better known) and the advantages of the other seem to be not large enough. I guess that's why ReLU is used most of the time. $\endgroup$– TinuCommented May 14, 2020 at 9:05
$\begingroup$
$\endgroup$
Leaky relu is a way to overcome the vanishing gradients buts as you increase the slope from 0 to 1 your activation function becomes linear, you can try to plot a leaky relu with different slopes in negative part. The problem is losing non-linearity with in cost of having a better gradient back propagation. If you can get a good result with relu, switching to leaky relu may result in getting worse.
$\begingroup$
$\endgroup$
Leaky ReLU can mitigate Dying ReLU Problem but 0 is still produced for the input value 0 so Dying ReLU Problem is not completely avoided.
And, I found Leaky ReLU is used in GAN(Generative Adversarial Network). *See DCGAN(Deep Convolutional Generative Adversarial Network) in PyTorch and DCGAN in TensorFlow.
