11
$\begingroup$

Batch normalization and ReLUs are both solutions to the vanishing gradient problem. If we're using batch normalization, should we then use sigmoids? Or are there features of ReLUs that make them worthwhile even when using batchnorm?

I suppose that the normalization done in batchnorm will send zero activations negative. Does that mean that batchnorm solves the "dead ReLU" problem?

But the continuous nature of tanh and logistic remain appealing. If I'm using batchnorm, will tanh work better than ReLU?

I'm sure that the answer depends. So, what has worked in your experience, and what are the salient features of your application?

$\endgroup$
1
  • $\begingroup$ Even if the paper suggests to use BatchNorm before the activation, it has been found in practice that better solutions are yield if BN is applied after. If I don't overlook something that should mean, that in the latter case, BN has no effect on the activation. But of course, it is an open question, if BN would work better when applied before and with another activation than ReLU. In my opinion, no. Because ReLU still has other advantages, such as a simpler derivation. But I am also curious. Maybe someone made experiences in this field. $\endgroup$
    – oezguensi
    Dec 20, 2018 at 2:57

2 Answers 2

3
$\begingroup$

See, the basic concept behind the batch-normalization is that (excerpt from a Medium article)-

We normalize our input layer by adjusting and scaling the activations. For example, when we have features from 0 to 1 and some from 1 to 1000, we should normalize them to speed up learning. If the input layer is benefiting from it, why not do the same thing also for the values in the hidden layers, that are changing all the time, and get 10 times or more improvement in the training speed.

Read the article here.

This is the reason why we use Batch-normalization. Now coming to your question, see as the output of sigmoid is constraints between 0 and 1, and that what is the motto of Batch-normalization. If we use Bach-normalization with sigmoid activation, then it will be constrained between sigmoid(0) to sigmoid(1), that is between 0.5 to 0.73 ~ $frac{1}/{(1+1/e)}$. But if we use ReLU with Batch-normalization then the output will be spread over 0 to 1 which is the good thing for us as finally, we want the output as much diverse as it can be. So I think ReLU will be the best choice among other activations.

$\endgroup$
1
  • 1
    $\begingroup$ When we apply batch norm, we not only normalize the hidden layer, we also denormalize it with learnable paramters. So your hypothesis that batch norm with sigmoid activation will constrain it to sigmoid(0) to sigmoid(1) is wrong $\endgroup$ Jul 22, 2020 at 13:59
1
$\begingroup$

madman answered your question regarding batch normalization correctly and let me answer your second part that how continuous functions may seem appealing but relu is better than all of them and this statement is not from my side MR. Hinton quoted it "we were dumb people who were using sigmoid as an activation function and it took 30 years for that realization to occur that without understanding its form its's never gonna let your neuron go in learning state its always saturating so is is it's derivative and he called himself and all others dumbfounded people".So choosing an activation function merely because it's continuous and not looking at how its gonna affect your neuron's firing potential like when you use sigmoid your network is bound to face the vanishing gradient problem because your activation function is itself "saturating" and you have to find the right balance between saturation(like sigmoid) and too much activation(don't use exponential) and that is where a linear activation like Relu finds the middle path in between and batchnorm is there to help you more.

Note: If you are studying neural nets I would advise you to think of neural nets as big and deep composite functions so to understand what works and why it works you need to understand how a neural net creates a manifold of data in some higher dimension "representing" that data in which the goodness of manifold depends on your choice of functions and how a function transforms the other functions output when given to it as input.

$\endgroup$
1
  • $\begingroup$ if we use batchnorm to the inputs to sigmoid function, then can this alleviate the saturation of sigmoid? If the normalised inputs are forced to sit around zero then they will have non-zero derivatives $\endgroup$
    – siegfried
    Jan 22, 2022 at 3:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.