1
$\begingroup$

Suppose there's a network with N hidden layers.

There are 2 cases:

  1. The network is deep
  2. The network is shallow

I've been wondering how N affects choosing the activation function.

Will it affect, for example, Sigmoid more than Leaky ReLU?

$\endgroup$

1 Answer 1

0
$\begingroup$

The number of hidden layers does not have a direct affect on the performance of the activation function. However, with more hidden layers, the probability for the vanishing gradient meaning that the gradient takes on very small values making it hard for the neural network to learn properly.

The sigmoid function is know to have a higher risk for the vanishing gradient in larger networks, which is why, ReLU or Leaky ReLU are preferred in larger architectures. So the effect is more indirect.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.