Looks like ReLU is better then sigmoid or tanh for deep neural networks from all aspects:

  • simple
  • more biologically plausible
  • no gradient to vanish
  • better performance
  • sparsity

And I see only one advantage of sigmoid/tanh: they are bounded. It means that your activations won't blow up as you keep training and your network parameters won't take off to the sky.

Why should not we forget about sigmoid/tanh for deep neural networks?

  • $\begingroup$ can you please explain what does "blow up activations" mean? $\endgroup$ – theateist May 21 '18 at 18:14
  • In certain network structures having symmetric activation layers has advantages (certain autoencoders for example)

  • In certain scenarios having an activation with mean 0 is important (so tanh makes sense).

  • Sigmoid activation in the output layer is still important for classification

In 95% of the cases ReLU is much better though.

  • $\begingroup$ What kind of advantages do symmetric activation layers have? Why is it not a disadvantage to have asymmetric activation layers? It seems odd to me that negative weights should be treated differently from positive weights. $\endgroup$ – asmaier Aug 22 '18 at 13:53

Sigmoid helps in controlling the activation unlike ReLu which blows up it up. Sigmoids don't overfit as much. Have a look at this.

I still would ask you to start with ReLu for training as it performs better most of the time.


I thought about three cases:

  1. Binary classification tasks: you can do it with one output node + Sigmoid + Binary Crossentropy loss.

  2. Multiple independent classifications: when an observation can be classified in two or more than two classes, and each class is independent from the others. In this case, even if you have many output nodes, the final activation function must be Sigmoid.

  3. LSTM and GRU layers contain Sigmoid and TanH gates. They have the right shape to let the RNN determine how much of past information must be forgotten/discarded/passed forward. (It's possible to change them with ReLU's, but there is no evidence this improves the inner performance of Recurrent layers.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.