Batch normalization variance calculation

Question

In batch normalization the variance calculation during the training phase is done by ($x_i$ are the individual elements in the training batch of size $m$)

$\sigma_B^2 = \frac 1m \sum_{i=1}^{m} (x_i - \mu_B)^2$
where $\mu_B = \frac 1m \sum_{i = 1}^{m}(x_i)$

and during the test time we calculate the population statistics for the same as

$E[x] = E_B[\mu_B]$ and $Var[x] = \frac{m}{m-1}E_B[\sigma_B^2]$ (as you can see the unbiased estimate of population variance is calculated during the test time as we are building the model to predict the population distribution)

Would it not seem more logical that bias correction be done during training phase itself and subsequently its expected value can be used without the $\frac {m}{m-1}$ correction factor. i.e

During training:

$\sigma_B^2 = \frac 1{m-1} \sum_{i=1}^{m} (x_i - \mu_B)^2$
where $\mu_B = \frac 1m \sum_{i = 1}^{m}(x_i)$

and during test time:

$E[x] = E_B[\mu_B]$ and $Var[x] = E_B[\sigma_B^2]$

Answer 1

@shimao has answered it Here on stats stackexchange

To quote shimao

"Yes, it would be more logical, although equivalent both ways. I was able to find a bit of discussion on this here

It looks like the original authors later implied that both training and test time should use the unbiased variance. Also the actual implementation in many libraries is not completely consistent with what is described in the paper."

Answer 2

Well, the only reason, I would say here, is the numerical stability. For instance, if the Batch size $m =1$ during the training, then you may end up with $NaN$ variance!

Update: I doubled check the BN paper again, they clearly mentioned the condition that $m>1$. However, I found an interesting post discussing the cases when biased estimator preferable than the unbiased one. This might explain why the authors used the biased one during the training and the unbiased one inference.