Looks like ReLU is better then sigmoid or tanh for deep neural networks from all aspects:
- more biologically plausible
- no gradient to vanish
- better performance
And I see only one advantage of sigmoid/tanh: they are bounded. It means that your activations won't blow up as you keep training and your network parameters won't take off to the sky.
Why should not we forget about sigmoid/tanh for deep neural networks?