2
$\begingroup$

I'm a beginner in NNs and the first thing I don't understand with batch norm is the following 2 steps:

  • First we normalize the batch data on a z parameter to Mu=0, sigma^2=1
  • Then we change z via the coefficients of Mu, sigma^2 (usu. alpha, beta) by updating them as learnable parameters.

I don't understand why the first step is necessary if we change the distribution in the second step anyway. Could someone explain please?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

The first step helps to reduce something called “internal covariate shift” of the network. Normalizing the layer inputs before applying the shift and scaling in step two, speeds up the training process (see BN paper).

This normalization comes with a cost, namely, it can reduce the number of possible representations a layer can provide. E.g. Normalized inputs of a sigmoid are constrained to the linear regime of the function. See BN paper on page three.

The second step is there to address this problem. Scaling and shifting the values to “not just” linear domains of the nonlinearities solves the representional issue while keeping the internal covariate shift at a minimum.

Improve this answer
$\endgroup$
0
1
$\begingroup$

The image says it all

I hope I got your question correctly...

It’s called “Batch” Normalization because we perform this transformation and calculate the statistics only for a subpart (a batch) of the entire training set not as a whole.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.