10
$\begingroup$

Batch Normalization is described in this paper as a normalization of the input to an activation function with scale and shift variables $\gamma$ and $\beta$. This paper mainly describes using the sigmoid activation function, which makes sense. However, it seems to me that feeding an input from the normalized distribution produced by the batch normalization into a ReLU activation function of $max(0,x)$ is risky if $\beta$ does not learn to shift most of the inputs past 0 such that the ReLU isn't losing input information. I.e. if the input to the ReLU were just standard normalized, we would lose a lot of our information below 0. Is there any guarantee or initialization of $\beta$ that will guarantee that we don't lose this information? Am I missing something with how the operation of BN and ReLU work?

$\endgroup$
3
$\begingroup$

That is known a problem with the ReLU activation functions. It is often called a "dying ReLU". Given an input over the zero boundary, the unit is now almost always closed. A closed ReLU cannot update its input parameters, a dead ReLU stays dead.

The solution is to use variants of ReLU for the activation function such as Leaky ReLU, Noisy ReLUs, or ELUs.

$\endgroup$
1
$\begingroup$

I'd say BN goes after the ReLU and not before, in general it should be put between 2 layers so to normalize the layer output PDF before becoming another layer input

The convolutive layer processing is composed of a Lin (Conv Operator) + NonLin (e.g. ReLU) processing (as the Artificial Neuron Processing) and a sparsifying nonlin like ReLU produces an output PDF which is non-negative as a result of filtering, so before passing it as the next layer input the BN can help renormalizing it

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.