8
$\begingroup$

In the paper Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift (here) Before explaining the process of batch normalization the paper tries to explain the issues related with (I am not getting what the exact issue addressed here is).

excerpt from section 2, para 2:

We could consider whitening activations at every training step or at some interval, either by modifying the network directly or by changing the parameters of the optimization algorithm to depend on the network activation values (Wiesler et al., 2014; Raiko et al., 2012; Povey et al., 2014; Desjardins & Kavukcuoglu). However, if these modifications are interspersed with the optimization steps, then the gradient descent step may attempt to update the parameters in a way that requires the normalization to be updated, which reduces the effect of the gradient step. For example, consider a layer with the input u that adds the learned bias $b$, and normalizes the result by subtracting the mean of the activation computed over the training data: $\hat x= x − E[x]$ where $x = u + b, X = {x_{1...N}}$ is the set of values of $x$ over the training set, and $E[x] = \frac 1 N(\sum_{i=1}^nx_i)$ .

If a gradient descent step ignores the dependence of E[x] on b, then it will update $b ← b + ∆b$, where $∆b ∝ −\partial l/\partial\hat x$. Then $$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$.

Thus, the combination of the update to b and subsequent change in normalization led to no change in the output of the layer nor, consequently, the loss. As the training continues, b will grow indefinitely while the loss remains fixed. This problem can get worse if the normalization not only centers but also scales the activations.

here is my understanding of the literature:

  1. We have a batch of size N (One training batch)

  2. Let there be two arbitrary hidden layer connected to each other (L1 and L2) connected by parameters $W$ and $b$

  3. output coming out of L1 is x1

  4. $u = x1W$ (this is where the literature above starts. dimension of u is MxN) (M is the number of units in L2)

  5. $x = u+b$ (dimension b = dimension x = dimension u = MxN)

  6. Now before feeding x into L2 we centre it by subtracting the mean of $x$ from each entry in $x$ ($\hat x= x − E[x]$)

  7. We compute the loss and backpropogate the gradient and update just this layer's $b$ in order to give it a sanity test. New $b$ = $b + \Delta b$

  8. We run it again on the same batch with updated $b$

  9. repeat 3 and 4

  10. $x_{new} = u+b + \Delta b$ (dimension b, $\Delta b$ = dimension x = dimension u = MxN)

  11. Now before feeding x into L2 we centre it by subtracting the mean of $x$ from each entry in $x$ ($\hat x = x + \Delta b − E[x + \Delta b] = x - E[x]$). which is the same as what was calculated before updating b and hence updating b had to effect on the training

My question is with this part of the excerpt:

"If a gradient descent step ignores the dependence of E[x] on b, then it will update $b ← b + ∆b$, where $∆b ∝ −\partial l/\partial\hat x$. Then $$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$."

Why is

" $$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$." dependent upon what comes before it? What is even the point of that bit? Please also note the useage of word "Then" (made bold) implying the statement necessarily draws causality from what comes before

$\endgroup$
3
$\begingroup$

Let's assume that you are trying to minimize the following loss for a given task, $$\ell(y, \hat{y}) = \frac{1}{2}\| y - \hat{y}\|^2, $$ where $\| \cdot\|$ is the Eucledian distance, and the predicted output is $\hat{y} =\hat{x}$ for simplicity. The gradients then can be computed as follow:

$$\Delta b = - \frac{\partial\ell}{\partial\hat{x}} \cdot \frac{\partial\hat{x}}{\partial b}, \hspace{20pt} \Delta \omega = - \frac{\partial\ell}{\partial\hat{x}} \cdot \frac{\partial\hat{x}}{\partial \omega}$$

Now, the gradient of $\hat{x}$ with respect to the bias $b$ is $$\frac{\partial\hat{x}}{\partial b} = \frac{\partial}{\partial b}(x-E[x]) = \frac{\partial}{\partial b}\left((u+b)-E[(u+b)]\right) = 1 - \frac{\partial}{\partial b}E[(u+b)]$$

Ignoring the fact $E(x)$ is dependant on $b$ makes the above gradient equals to 1, and hence keeps updating the bias as follow:

$$\frac{\partial\hat{x}}{\partial b} = 1 - \frac{\partial}{\partial b}E[x] = 1 -0 = 1$$ and then

$$\Delta b = - \frac{\partial\ell}{\partial\hat{x}} \cdot (1), \hspace{20pt} b \leftarrow b + \Delta b $$

Otherwise, if you wanna consider this dependency, the gradient becomes 0, and hence no update as follow:

$$\frac{\partial\hat{x}}{\partial b} = 1 - \frac{\partial}{\partial b}E[u+b] = 1 - \left(\frac{\partial}{\partial b}E[u]+\frac{\partial}{\partial b}E[b]\right) = 1 - (0+1) = 0$$ and then $$\Delta b = - \frac{\partial\ell}{\partial\hat{x}} \cdot (0), \hspace{20pt} b \leftarrow b + 0 $$

In both cases, regardless bias update, the loss function will remain fixed,

$$u+(b+\Delta b)−E[u+(b+∆b)]=u+b−E[u+b], $$

however, in the first case, the bias will grow indefinitely.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.