How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

Question

How do you use LeakyRelu as an activation function in sequence DNN in keras? If I want to write something similar to:

 model = Sequential()
    model.add(Dense(90, activation='LeakyRelu'))

What is the solution? Put LeakyRelu similar to Relu?

Second question is: what are the best general setting for tuning the parameters of LeakyRelu? When is its performance significantly better than Relu?

Answer 1

You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer.

Import the LeakyReLU and instantiate a model

from keras.layers import LeakyReLU
model = Sequential()

# here change your line to leave out an activation 
model.add(Dense(90))

# now add a ReLU layer explicitly:
model.add(LeakyReLU(alpha=0.05))

---

Being able to simply write e.g. activation='relu' is made possible because of simple aliases that are created in the source code.

---

For your second question:

what are the best general setting for tuning the parameters of LeakyRelu? And when its performance is significantly better than Relu?

I can't give you optimal settings for the LeakyReLU, I'm afraid - they will be model/data dependent.

The difference between the ReLU and the LeakyReLU is the ability of the latter to retain some degree of the negative values that flow into it, whilst the former simply sets all values less than 0 to be 0. In theory, this extended output range offers a slightly higher flexibility to the model using it. I'm sure the inventors thought it to be useful and perhaps proved that to be the case for a few benchmarks. In practice, however, people generally just stick to the ReLU, as the benefits of the LeakyReLU are not consistent and the ReLU is cheaper to compute and therefore models train slightly faster.

Answer 2

I believe the question was about using LeayReLU within the Keras Functional API. Which would look something like this:

from keras.layers import LeakyReLU
...
x = Dense(128)(x)
x = LeakyReLU(alpha=0.3)(x)

```

Answer 3

Just to add another solution.
You can also write something like

import tensorflow as tf
keras = tf.keras  

layer1 = keras.layers.Dense(units=90, activation=keras.layers.LeakyReLU(alpha=0.01))
model = keras.Sequential([layer1])

or

model= keras.Sequential([                                 
          keras.layers.Dense(units=90, 
              activation=keras.layers.LeakyReLU(alpha=0.01))                         
])

However, passing 'advanced activation' layers through the 'activation' argument of a layer is not a good practice and is best to be avoided. Refer to the Official Docs for more -
Layer Activation Functions

Answer 4

LeakyRelu allows a small gradient when the unit is not active (negative):

$$ f(x) = alpha * x \:\text{for} \: x < 0, $$ $$f(x) = x \:\text{for}\: x >= 0.$$