As leaky relu does not lead any value to 0, so training always continues. And I can't think of any disadvantages it have.
Yet Leaky relu is less popular than Relu in real practice. Can someone tell why?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
Speaking from my experience, the performance of the two is almost the same. It might depend on the problem.
LeakyReLU was introduced to address the vanishing gradient problem, however it introduces yet another hyperparameter, the slope. If you want to squeeze out a little bit more performance of your model you can use LeakyReLU and tune the slope parameter, but that comes again of the cost of potential overfitting.
Leaky relu is a way to overcome the vanishing gradients buts as you increase the slope from 0 to 1 your activation function becomes linear, you can try to plot a leaky relu with different slopes in negative part. The problem is losing non-linearity with in cost of having a better gradient back propagation. If you can get a good result with relu, switching to leaky relu may result in getting worse.