The state of the art of nonlinearity is to use rectified linear units (ReLU) instead of a sigmoid function in deep neural networks. What are the advantages?
$\begingroup$
$\endgroup$
asked Jul 22, 2017 at 17:56
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
The sigmoid function becomes asymptotically either zero or one which means that the gradients are near zero for inputs with a large absolute value.
This makes the sigmoid function prone to vanishing gradient issues which the ReLU does not suffer as much.
In addition, ReLU has an attribute which can be seen both as positive and negative depending on which angle you are approaching it. The fact that ReLU is effectively a function that is zero for negative inputs and identity for positive inputs means that it is easy to have zeros as outputs and this leads to dead neurons. However, dead neurons might sound bad but in many cases, it is not because it allows for sparsity. In a way, the ReLU does a similar job of what an L1 regularization would do which would bring some weights to zero which in turn means a sparse solution.
Sparsity is something that, lots of times, leads to a better generalization of the model but there are times which has a negative impact on performance so it depends.
A good practice when using ReLU is to initialize the bias to a small number rather than zero so that you avoid dead neurons at the beginning of the training of the neural network which might prevent training in general.
answered Jul 22, 2017 at 17:56
-
$\begingroup$ So the only undisputed advantage of ReLU is non-saturation? $\endgroup$– AlexAug 8, 2017 at 23:28
$\begingroup$
$\endgroup$
3
To add to George Pligor's comment, it is a good idea to use Xavier weight initialization while using ReLU. Here is a description of this idea: http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization
answered Jul 22, 2017 at 18:22
-
1$\begingroup$ Actually, that is very subjective and, according to the latest state of the art, not that much of a good idea. In He et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification" (link), the authors proved that a zero-mean Gaussian distribution of variance
2/n_inputs(later on called "varscaling" in some deep learning libraries) performs better than Xavier initialization. $\endgroup$– E_net4Jul 22, 2017 at 22:32 -
$\begingroup$ Thank you for the info, didn't know. But the authors use PReLU, is their weight intialization relevant in case of simple ReLU? $\endgroup$ Jul 23, 2017 at 4:44
-
$\begingroup$ Yes, it still is. You can find the comparison in their paper (also have a look at Figure 3). $\endgroup$– E_net4Jul 23, 2017 at 9:46