As leaky relu does not lead any value to 0, so training always continues. And I can't think of any disadvantages it have.
Yet Leaky relu is less popular than Relu in real practice. Can someone tell why?
As leaky relu does not lead any value to 0, so training always continues. And I can't think of any disadvantages it have.
Yet Leaky relu is less popular than Relu in real practice. Can someone tell why?
Speaking from my experience, the performance of the two is almost the same. It might depend on the problem.
LeakyReLU was introduced to address the vanishing gradient problem, however it introduces yet another hyperparameter, the slope. If you want to squeeze out a little bit more performance of your model you can use LeakyReLU and tune the slope parameter, but that comes again of the cost of potential overfitting.