# Heuristic argument for Weight decay and regularization

I have been reading this book and I have no idea how to make an argument for this problem. http://neuralnetworksanddeeplearning.com/chap3.html

Connecting regularization and the improved method of weight initialization L2 regularization sometimes automatically gives us something similar to the new approach to weight initialization.

Suppose we are using the old approach to weight initialization. Sketch a heuristic argument that:

(1) supposing $λ$ is not too small, the first epochs of training will be dominated almost entirely by weight decay;

(2) provided $ηλ≪n$ the weights will decay by a factor of $e^{\frac{−ηλ}{m}}$ per epoch; and

(3) supposing λ is not too large, the weight decay will tail off when the weights are down to a size around $\frac{1}{\sqrt{n}}$, where n is the total number of weights in the network. Argue that these conditions are all satisfied in the examples graphed in this section.

Can anyone explain a little bit more and guide my direction? Thank you so much for reading it.

My argument is that:

(1) I am thinking about this formula $w' = w(1-\frac{ηλ}{n}) - \frac{n}{m} \sum \frac{dCx}{dw}$ is related. Since the weight is larger without using new weight initialization (the standard deviation of $w = \frac{1}{\sqrt{total number of weights in the network}}$, $\frac{ηλw}{n}$ will be significantly larger than sum of $\frac{dCx}{dw}$ （the average rate of change of $C$ in respect of weight for a minibatch).

(2) By substituting $ηλ+b = n$ into $w'$??

(3)

Thank you again.

Perhaps I can answer the first and second one:

(1)According to (93) on his book: $w \rightarrow w=\left(1-\frac{\eta \lambda}{n}\right) w -\frac{\eta}{m} \sum_x \frac{\partial C_x}{\partial w}$, there is an item $(1-\frac{\eta \lambda}{n})$on the weight, so weights will shrink every epoch. As weights were random chose at the beginning, it seems reasonable that the changes(the item $\frac{\eta}{m}\sum_x\frac{\partial C_x}{\partial w}$) will be chaotic on first epoch. Maybe that is why the weight decay will dominate the first epoch. And there is another explanation here, which may be helpful.

(2)At each epoch, the item $(1-\frac{\eta \lambda}{n})$ will be multiplied $\frac{n}{m}$times, where $n$ is the number of training data, $m$ is the size of minibatch. So after each epoch, the weight will decay by a factor of$(1-\frac{\eta \lambda}{n})^\frac{n}{m}$. According to the definition of $e$:$e=\lim_{n\to \infty}(1+\frac{1}{n})^n$, lets change the format a little bit:

$(1-\frac{\eta \lambda}{n})^\frac{n}{m}=[(1-\frac{\eta \lambda}{n})^{-\frac{n}{\eta \lambda}}]^{-\frac{\eta \lambda}{m}}$

As you can see, when $\eta \lambda \ll n$, $(1-\frac{\eta \lambda}{n})^{-\frac{n}{\eta \lambda}} \approx e$.Then we can come to the conclusion that provided $ηλ≪n$ the weights will decay by a factor of $exp(−ηλ/m)$ per epoch.

(3)I have been through this for two days and have no idea about it,but I think it should be related to the formula(85):

$\begin{eqnarray} C = -\frac{1}{n} \sum_{xj} \left[ y_j \ln a^L_j+(1-y_j) \ln (1-a^L_j)\right] + \frac{\lambda}{2n} \sum_w w^2. \end{eqnarray}$

where the weights were summed on last item.

• After months of confusion, I almost forgot the question. Thanks! For question 3, I think it deserves a separate thread maybe. – BenjiBB Jul 22 '17 at 16:46

$$C = C_0 + {\lambda \over 2n} \sum_{i=1}^{N_w}w^2$$ $$C = {\lambda N_w w^2 \over 2n}$$ $$W = \sqrt{2nC \over \lambda} {1 \over \sqrt{N_w}}$$ $$W \propto {1 \over \sqrt{N_w}}$$

Here you go for question 3.

• Any reason you don;t just write that up? – Stephen Rauch Apr 1 '18 at 0:02