Using LeakyRelu as activation function in CNN and best alpha for it

Question

Since if we do not declare the activation function, the default will be set as linear for Conv2D layer. Is it true to write:

model.add(Conv2D(32, kernel_size=(3, 3),
           input_shape=(380,380,1))
model.add(LeakyReLU(alpha=0.01))

I mean now by the written lines, the activation function for Conv2D layer is set as LeakyRelu or not?

Further, I want to know what is the best alpha? I couldn't find any resources analyzing it.

Answer 1

The code,

model.add(Conv2D(32, kernel_size=(3, 3), input_shape=(380,380,1))
model.add(LeakyReLU(alpha=0.01))

will definitely transform the outputs from the Conv2D using the LeakyReLU activation given parameter alpha ( negative slope of ReLU ).

Further, I want to know what is the best alpha? I couldn't find any resources analyzing it.

In ReLU, we simply set the activation to 0 for negative values. This causes the dying ReLU problem which leds to overfitting. Hence, we return $x \alpha$ instead of 0, so that the unit does not become non-functional.

If we look at TensorFlow's tf.nn.leaky_relu method, we will find that the alpha is 0.2.

Whereas in Keras' layers.LeakyReLU class, you will find the alpha is 0.3.

So you can clearly get an idea of what the parameter's value should be. It's basically a hyperparameter which you need to adjust through trial-error observations.