As leaky relu does not lead any value to 0, so training always continues. And I can't think of any disadvantages it have.
Yet Leaky relu is less popular than Relu in real practice. Can someone tell why?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
Speaking from my experience, the performance of the two is almost the same. It might depend on the problem.
LeakyReLU was introduced to address the vanishing gradient problem, however it introduces yet another hyperparameter, the slope. If you want to squeeze out a little bit more performance of your model you can use LeakyReLU and tune the slope parameter, but that comes again of the cost of potential overfitting.