0
$\begingroup$

I read the article Batch normalization: theory and how to use it with Tensorflow by Federico Peccia.

The batch normalized activation is $$ \bar x_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} $$ where $\mu_B = \frac{1}{m} \sum_{i=1}^m x_i$ is the batch mean and $\sigma_B^2 = \frac{1}{m} \sum_{i=1}^m (x_i - \mu_B)^2$ is the batch variance. The scaled and shifted activation is $y_i = \gamma \bar x_i + \beta$ where $\gamma$ and $\beta$ are parameters that the neural network learns.

After these definition the following set of equations appears: enter image description here

I think there are mistakes in it.

Suppose there are $j$ batches, each of size $m$. I think the first equation (inference mean) is "average of means" so it should be $$ E_x = \frac{1}{j} \sum_{i=1}^{j} \mu_B^{(i)} $$ because this is the mean of $j$ mini-batches.

Similarly in the second equation. What are the correct formulas?

In the third equation, shouldn't it be $$ y = \gamma \bar x + \beta = \gamma \frac{x - E_x}{\sqrt{\mathrm{Var}_x + \epsilon}} + \beta = \frac{\gamma}{\sqrt{\mathrm{Var}_x + \epsilon}} x + \beta - \frac{\gamma E_x}{\sqrt{\mathrm{Var}_x + \epsilon}} ? $$

Wikipedia (Batch notmalization) confirms my "debugging".

$\endgroup$

1 Answer 1

1
$\begingroup$

According to the paper "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" you are correct. m is the batch size number and not the number of total mini-batches. So the division is by j. In addition, when the variables are placed in the third formula, the equation you formulate is indeed obtained.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.