Batch normalization and ReLUs are both solutions to the vanishing gradient problem. If we're using batch normalization, should we then use sigmoids? Or are there features of ReLUs that make them worthwhile even when using batchnorm?
I suppose that the normalization done in batchnorm will send zero activations negative. Does that mean that batchnorm solves the "dead ReLU" problem?
But the continuous nature of tanh and logistic remain appealing. If I'm using batchnorm, will tanh work better than ReLU?
I'm sure that the answer depends. So, what has worked in your experience, and what are the salient features of your application?