# How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

How do you use LeakyRelu as an activation function in sequence DNN in keras? If I want to write something similar to:

 model = Sequential()


What is the solution? Put LeakyRelu similar to Relu?

Second question is: what are the best general setting for tuning the parameters of LeakyRelu? When is its performance significantly better than Relu?

You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer.

Import the LeakyReLU and instantiate a model

from keras.layers import LeakyReLU
model = Sequential()

# here change your line to leave out an activation

# now add a ReLU layer explicitly:


Being able to simply write e.g. activation='relu' is made possible because of simple aliases that are created in the source code.

what are the best general setting for tuning the parameters of LeakyRelu? And when its performance is significantly better than Relu?

I can't give you optimal settings for the LeakyReLU, I'm afraid - they will be model/data dependent.

The difference between the ReLU and the LeakyReLU is the ability of the latter to retain some degree of the negative values that flow into it, whilst the former simply sets all values less than 0 to be 0. In theory, this extended output range offers a slightly higher flexibility to the model using it. I'm sure the inventors thought it to be useful and perhaps proved that to be the case for a few benchmarks. In practice, however, people generally just stick to the ReLU, as the benefits of the LeakyReLU are not consistent and the ReLU is cheaper to compute and therefore models train slightly faster.

• if we plot or print summary of the model we see that there are 2 layers (first layer of dense with 90 units) and another layer of leakyrelu. Is there a way to create one layer (same as relu) ? – Boom Jun 2 at 4:19

I believe the question was about using LeayReLU within the Keras Functional API. Which would look something like this:

from keras.layers import LeakyReLU
...
x = Dense(128)(x)
x = LeakyReLU(alpha=0.3)(x)

$$$$

• if we plot or print summary of the model we see that there are 2 layers (first layer of dense with 128 units) and another layer of leakyrelu. Is there a way to create one layer (same as relu) – Boom Jun 2 at 4:23

You can also write something like

import tensorflow as tf
keras = tf.keras


layer1 = keras.layers.Dense(units=90, activation=keras.layers.LeakyReLU(alpha=0.01))
model = keras.Sequential([layer1])


or

model= keras.Sequential([
keras.layers.Dense(units=90,
activation=keras.layers.LeakyReLU(alpha=0.01))
])
`

However, passing 'advanced activation' layers through the 'activation' argument of a layer is not a good practice and is best to be avoided. Refer to the Official Docs for more -
Layer Activation Functions

LeakyRelu allows a small gradient when the unit is not active (negative):

$$f(x) = alpha * x \:\text{for} \: x < 0,$$ $$f(x) = x \:\text{for}\: x >= 0.$$