Skip to main content
updated to provide an answer for the second question
Source Link
n1k31t4
  • 15.1k
  • 2
  • 31
  • 51

You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer.

Import the LeakyReLU and instantiate a model

from keras.layers import LeakyReLU
model = Sequential()

# here change your line to leave out an activation 
model.add(Dense(90))

# now add a ReLU layer explicitly:
model.add(LeakyReLU(alpha=0.05))

Being able to simply write e.g. activation='relu' is made possible because of simple aliases that are created in the source code.


For your second question:

what are the best general setting for tuning the parameters of LeakyRelu? And when its performance is significantly better than Relu?

I can't give you optimal settings for the LeakyReLU, I'm afraid - they will be model/data dependent.

The difference between the ReLU and the LeakyReLU is the ability of the latter to retain some degree of the negative values that flow into it, whilst the former simply sets all values less than 0 to be 0. In theory, this extended output range offers a slightly higher flexibility to the model using it. I'm sure the inventors thought it to be useful and perhaps proved that to be the case for a few benchmarks. In practice, however, people generally just stick to the ReLU, as the benefits of the LeakyReLU are not consistent and the ReLU is cheaper to compute and therefore models train slightly faster.

You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer.

Import the LeakyReLU and instantiate a model

from keras.layers import LeakyReLU
model = Sequential()

# here change your line to leave out an activation 
model.add(Dense(90))

# now add a ReLU layer explicitly:
model.add(LeakyReLU(alpha=0.05))

Being able to simply write e.g. activation='relu' is made possible because of simple aliases that are created in the source code.

You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer.

Import the LeakyReLU and instantiate a model

from keras.layers import LeakyReLU
model = Sequential()

# here change your line to leave out an activation 
model.add(Dense(90))

# now add a ReLU layer explicitly:
model.add(LeakyReLU(alpha=0.05))

Being able to simply write e.g. activation='relu' is made possible because of simple aliases that are created in the source code.


For your second question:

what are the best general setting for tuning the parameters of LeakyRelu? And when its performance is significantly better than Relu?

I can't give you optimal settings for the LeakyReLU, I'm afraid - they will be model/data dependent.

The difference between the ReLU and the LeakyReLU is the ability of the latter to retain some degree of the negative values that flow into it, whilst the former simply sets all values less than 0 to be 0. In theory, this extended output range offers a slightly higher flexibility to the model using it. I'm sure the inventors thought it to be useful and perhaps proved that to be the case for a few benchmarks. In practice, however, people generally just stick to the ReLU, as the benefits of the LeakyReLU are not consistent and the ReLU is cheaper to compute and therefore models train slightly faster.

Source Link
n1k31t4
  • 15.1k
  • 2
  • 31
  • 51

You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer.

Import the LeakyReLU and instantiate a model

from keras.layers import LeakyReLU
model = Sequential()

# here change your line to leave out an activation 
model.add(Dense(90))

# now add a ReLU layer explicitly:
model.add(LeakyReLU(alpha=0.05))

Being able to simply write e.g. activation='relu' is made possible because of simple aliases that are created in the source code.