4
$\begingroup$

In batch normalization the variance calculation during the training phase is done by ($x_i$ are the individual elements in the training batch of size $m$)

$\sigma_B^2 = \frac 1m \sum_{i=1}^{m} (x_i - \mu_B)^2$
where $\mu_B = \frac 1m \sum_{i = 1}^{m}(x_i)$

and during the test time we calculate the population statistics for the same as

$E[x] = E_B[\mu_B]$ and $Var[x] = \frac{m}{m-1}E_B[\sigma_B^2]$ (as you can see the unbiased estimate of population variance is calculated during the test time as we are building the model to predict the population distribution)

Would it not seem more logical that bias correction be done during training phase itself and subsequently its expected value can be used without the $\frac {m}{m-1}$ correction factor. i.e

During training:

$\sigma_B^2 = \frac 1{m-1} \sum_{i=1}^{m} (x_i - \mu_B)^2$
where $\mu_B = \frac 1m \sum_{i = 1}^{m}(x_i)$

and during test time:

$E[x] = E_B[\mu_B]$ and $Var[x] = E_B[\sigma_B^2]$

$\endgroup$
1
$\begingroup$

Well, the only reason, I would say here, is the numerical stability. For instance, if the Batch size $m =1$ during the training, then you may end up with $NaN$ variance!

Update: I doubled check the BN paper again, they clearly mentioned the condition that $m>1$. However, I found an interesting post discussing the cases when biased estimator preferable than the unbiased one. This might explain why the authors used the biased one during the training and the unbiased one inference.

| improve this answer | |
$\endgroup$
  • $\begingroup$ If your batch size is 1, then batch normalisation would be a terrible choice, all training features would be 0 for all examples! $\endgroup$ – Neil Slater Nov 1 '17 at 13:57
  • $\begingroup$ @NeilSlater Totally agree! Actually, I have double checked the BN paper, and they clearly mentioned the condition that $m > 1$. $\endgroup$ – Shadi Nov 1 '17 at 16:33
  • $\begingroup$ During training we have backpropogated using biased estimator and during test we are using the unbiased estimator. If we had used the unbiased estimator during the training itself then it fits into a more robust cohesive whole where model was trained on the same conditions that the model is tested on. this is bugging me and my ocd badly now haha $\endgroup$ – MiloMinderbinder Nov 1 '17 at 19:15
  • $\begingroup$ Please refer to shimao's answer : stats.stackexchange.com/questions/311074/… $\endgroup$ – MiloMinderbinder Nov 3 '17 at 12:07
1
$\begingroup$

@shimao has answered it Here on stats stackexchange

To quote shimao

"Yes, it would be more logical, although equivalent both ways. I was able to find a bit of discussion on this here

It looks like the original authors later implied that both training and test time should use the unbiased variance. Also the actual implementation in many libraries is not completely consistent with what is described in the paper."

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.