Many paper and books say that sigmoid activation function with random intialization is prone to vanishing/exploding gradients therefore it is better to use LeakyRelu, Elu, or Relu. Does this mean that we should use them in final layer of binary classificiation as well?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Vanishing & Exploding Gradient problem happens in case of deep neural network. In NN when we have to update weights & biases for each layer we calculate the partial derivate with respect to y_hat at each layer (Back Propogation Algorithm). Because in this case weights are multiplied in chain with each other.
As Sigmoid is used in last layer it will only be just gradient and does not have impact of other layer while initial layer will be multiplies by weights of earlier layer leading to Vanishing Gradient problem.
So Sigmoid in last layer does not lead to Vanishing Gradient problem and you can use it safely.
$\begingroup$
$\endgroup$
There's no problem using the sigmoid function in the final layer. Vanishing/exploding gradients only become a issue when the sigmoid function is used across multiple layers.
Sigmoid is used "behind the scenes" when you use the softmax function for multi-class classification. A sigmoid function is used for each class label, these are then normalised so the class probabilities across all labels sum to 1.
