# Does batch normalization mean that sigmoids work better than ReLUs?

Batch normalization and ReLUs are both solutions to the vanishing gradient problem. If we're using batch normalization, should we then use sigmoids? Or are there features of ReLUs that make them worthwhile even when using batchnorm?

I suppose that the normalization done in batchnorm will send zero activations negative. Does that mean that batchnorm solves the "dead ReLU" problem?

But the continuous nature of tanh and logistic remain appealing. If I'm using batchnorm, will tanh work better than ReLU?

I'm sure that the answer depends. So, what has worked in your experience, and what are the salient features of your application?

• Even if the paper suggests to use BatchNorm before the activation, it has been found in practice that better solutions are yield if BN is applied after. If I don't overlook something that should mean, that in the latter case, BN has no effect on the activation. But of course, it is an open question, if BN would work better when applied before and with another activation than ReLU. In my opinion, no. Because ReLU still has other advantages, such as a simpler derivation. But I am also curious. Maybe someone made experiences in this field. – oezguensi Dec 20 '18 at 2:57

See, the basic concept behind the batch-normalization is that (excerpt from a Medium article)-

We normalize our input layer by adjusting and scaling the activations. For example, when we have features from 0 to 1 and some from 1 to 1000, we should normalize them to speed up learning. If the input layer is benefiting from it, why not do the same thing also for the values in the hidden layers, that are changing all the time, and get 10 times or more improvement in the training speed.

This is the reason why we use Batch-normalization. Now coming to your question, see as the output of sigmoid is constraints between 0 and 1, and that what is the motto of Batch-normalization. If we use Bach-normalization with sigmoid activation, then it will be constrained between sigmoid(0) to sigmoid(1), that is between 0.5 to 0.73 ~ $$frac{1}/{(1+1/e)}$$. But if we use ReLU with Batch-normalization then the output will be spread over 0 to 1 which is the good thing for us as finally, we want the output as much diverse as it can be. So I think ReLU will be the best choice among other activations.

• When we apply batch norm, we not only normalize the hidden layer, we also denormalize it with learnable paramters. So your hypothesis that batch norm with sigmoid activation will constrain it to sigmoid(0) to sigmoid(1) is wrong – abhishek jha Jul 22 at 13:59