Looks like ReLU is better then sigmoid or tanh for deep neural networks from all aspects:
And I see only one advantage of sigmoid/tanh: they are bounded. It means that your activations won't blow up as you keep training and your network parameters won't take off to the sky.
Why should not we forget about sigmoid/tanh for deep neural networks?
In certain network structures having symmetric activation layers has advantages (certain autoencoders for example)
In certain scenarios having an activation with mean 0 is important (so tanh makes sense).
Sigmoid activation in the output layer is still important for classification
In 95% of the cases ReLU is much better though.
Sigmoid helps in controlling the activation unlike ReLu which blows up it up. Sigmoids don't overfit as much. Have a look at this.
I still would ask you to start with ReLu for training as it performs better most of the time.
I thought about three cases:
Binary classification tasks: you can do it with one output node + Sigmoid + Binary Crossentropy loss.
Multiple independent classifications: when an observation can be classified in two or more than two classes, and each class is independent from the others. In this case, even if you have many output nodes, the final activation function must be Sigmoid.
LSTM and GRU layers contain Sigmoid and TanH gates. They have the right shape to let the RNN determine how much of past information must be forgotten/discarded/passed forward. (It's possible to change them with ReLU's, but there is no evidence this improves the inner performance of Recurrent layers.)
