How do you use LeakyRelu as an activation function in sequence DNN in keras? If I want to write something similar to:

 model = Sequential()
    model.add(Dense(90, activation='LeakyRelu'))

What is the solution? Put LeakyRelu similar to Relu?

Second question is: what are the best general setting for tuning the parameters of LeakyRelu? When is its performance significantly better than Relu?


You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer.

Import the LeakyReLU and instantiate a model

from keras.layers import LeakyReLU
model = Sequential()

# here change your line to leave out an activation 

# now add a ReLU layer explicitly:

Being able to simply write e.g. activation='relu' is made possible because of simple aliases that are created in the source code.

For your second question:

what are the best general setting for tuning the parameters of LeakyRelu? And when its performance is significantly better than Relu?

I can't give you optimal settings for the LeakyReLU, I'm afraid - they will be model/data dependent.

The difference between the ReLU and the LeakyReLU is the ability of the latter to retain some degree of the negative values that flow into it, whilst the former simply sets all values less than 0 to be 0. In theory, this extended output range offers a slightly higher flexibility to the model using it. I'm sure the inventors thought it to be useful and perhaps proved that to be the case for a few benchmarks. In practice, however, people generally just stick to the ReLU, as the benefits of the LeakyReLU are not consistent and the ReLU is cheaper to compute and therefore models train slightly faster.


LeakyRelu allows a small gradient when the unit is not active (negative):

$$ f(x) = alpha * x \:\text{for} \: x < 0, $$ $$f(x) = x \:\text{for}\: x >= 0.$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.