Does batch normalization mean that sigmoids work better than ReLUs?

Question

Batch normalization and ReLUs are both solutions to the vanishing gradient problem. If we're using batch normalization, should we then use sigmoids? Or are there features of ReLUs that make them worthwhile even when using batchnorm?

I suppose that the normalization done in batchnorm will send zero activations negative. Does that mean that batchnorm solves the "dead ReLU" problem?

But the continuous nature of tanh and logistic remain appealing. If I'm using batchnorm, will tanh work better than ReLU?

I'm sure that the answer depends. So, what has worked in your experience, and what are the salient features of your application?

Answer 1

See, the basic concept behind the batch-normalization is that (excerpt from a Medium article)-

We normalize our input layer by adjusting and scaling the activations. For example, when we have features from 0 to 1 and some from 1 to 1000, we should normalize them to speed up learning. If the input layer is benefiting from it, why not do the same thing also for the values in the hidden layers, that are changing all the time, and get 10 times or more improvement in the training speed.

Read the article here.

This is the reason why we use Batch-normalization. Now coming to your question, see as the output of sigmoid is constraints between 0 and 1, and that what is the motto of Batch-normalization. If we use Bach-normalization with sigmoid activation, then it will be constrained between sigmoid(0) to sigmoid(1), that is between 0.5 to 0.73 ~ $frac{1}/{(1+1/e)}$. But if we use ReLU with Batch-normalization then the output will be spread over 0 to 1 which is the good thing for us as finally, we want the output as much diverse as it can be. So I think ReLU will be the best choice among other activations.

Answer 2

madman answered your question regarding batch normalization correctly and let me answer your second part that how continuous functions may seem appealing but relu is better than all of them and this statement is not from my side MR. Hinton quoted it "we were dumb people who were using sigmoid as an activation function and it took 30 years for that realization to occur that without understanding its form its's never gonna let your neuron go in learning state its always saturating so is is it's derivative and he called himself and all others dumbfounded people".So choosing an activation function merely because it's continuous and not looking at how its gonna affect your neuron's firing potential like when you use sigmoid your network is bound to face the vanishing gradient problem because your activation function is itself "saturating" and you have to find the right balance between saturation(like sigmoid) and too much activation(don't use exponential) and that is where a linear activation like Relu finds the middle path in between and batchnorm is there to help you more.

Note: If you are studying neural nets I would advise you to think of neural nets as big and deep composite functions so to understand what works and why it works you need to understand how a neural net creates a manifold of data in some higher dimension "representing" that data in which the goodness of manifold depends on your choice of functions and how a function transforms the other functions output when given to it as input.