Heuristic argument for Weight decay and regularization

I have been reading this book and I have no idea how to make an argument for this problem. http://neuralnetworksanddeeplearning.com/chap3.html

Connecting regularization and the improved method of weight initialization L2 regularization sometimes automatically gives us something similar to the new approach to weight initialization.

Suppose we are using the old approach to weight initialization. Sketch a heuristic argument that:

(1) supposing $λ$ is not too small, the first epochs of training will be dominated almost entirely by weight decay;

(2) provided $ηλ≪n$ the weights will decay by a factor of $e^{\frac{−ηλ}{m}}$ per epoch; and

(3) supposing λ is not too large, the weight decay will tail off when the weights are down to a size around $\frac{1}{\sqrt{n}}$, where n is the total number of weights in the network. Argue that these conditions are all satisfied in the examples graphed in this section.

Can anyone explain a little bit more and guide my direction? Thank you so much for reading it.

My argument is that:

(1) I am thinking about this formula $w' = w(1-\frac{ηλ}{n}) - \frac{n}{m} \sum \frac{dCx}{dw}$ is related. Since the weight is larger without using new weight initialization (the standard deviation of $w = \frac{1}{\sqrt{total number of weights in the network}}$, $\frac{ηλw}{n}$ will be significantly larger than sum of $\frac{dCx}{dw}$ （the average rate of change of $C$ in respect of weight for a minibatch).

(2) By substituting $ηλ+b = n$ into $w'$??

(3)

Thank you again.

Perhaps I can answer the first and second one:

(1)According to (93) on his book: $w \rightarrow w=\left(1-\frac{\eta \lambda}{n}\right) w -\frac{\eta}{m} \sum_x \frac{\partial C_x}{\partial w}$, there is an item $(1-\frac{\eta \lambda}{n})$on the weight, so weights will shrink every epoch. As weights were random chose at the beginning, it seems reasonable that the changes(the item $\frac{\eta}{m}\sum_x\frac{\partial C_x}{\partial w}$) will be chaotic on first epoch. Maybe that is why the weight decay will dominate the first epoch. And there is another explanation here, which may be helpful.

(2)At each epoch, the item $(1-\frac{\eta \lambda}{n})$ will be multiplied $\frac{n}{m}$times, where $n$ is the number of training data, $m$ is the size of minibatch. So after each epoch, the weight will decay by a factor of$(1-\frac{\eta \lambda}{n})^\frac{n}{m}$. According to the definition of $e$:$e=\lim_{n\to \infty}(1+\frac{1}{n})^n$, lets change the format a little bit:

$(1-\frac{\eta \lambda}{n})^\frac{n}{m}=[(1-\frac{\eta \lambda}{n})^{-\frac{n}{\eta \lambda}}]^{-\frac{\eta \lambda}{m}}$

As you can see, when $\eta \lambda \ll n$, $(1-\frac{\eta \lambda}{n})^{-\frac{n}{\eta \lambda}} \approx e$.Then we can come to the conclusion that provided $ηλ≪n$ the weights will decay by a factor of $exp(−ηλ/m)$ per epoch.

(3)I have been through this for two days and have no idea about it,but I think it should be related to the formula(85):

$\begin{eqnarray} C = -\frac{1}{n} \sum_{xj} \left[ y_j \ln a^L_j+(1-y_j) \ln (1-a^L_j)\right] + \frac{\lambda}{2n} \sum_w w^2. \end{eqnarray}$

where the weights were summed on last item.

• After months of confusion, I almost forgot the question. Thanks! For question 3, I think it deserves a separate thread maybe. Jul 22, 2017 at 16:46

$$C = C_0 + {\lambda \over 2n} \sum_{i=1}^{N_w}w^2$$ $$C = {\lambda N_w w^2 \over 2n}$$ $$W = \sqrt{2nC \over \lambda} {1 \over \sqrt{N_w}}$$ $$W \propto {1 \over \sqrt{N_w}}$$

Here you go for question 3.

• Any reason you don;t just write that up? Apr 1, 2018 at 0:02

I know it has been a few years since this thread has been active, but I would like to give my answer for part 3 of the question. I have been reading the same resource, and originally came here for an answer.

The book gives the statement to be shown as "The weight decay will tail off when the weights are down to a size around 1/√n ."

I think the best way to (heuristically) show this is via the definition of the altered cost function:

$C&space;=&space;C_{0}&space;+&space;\frac{\lambda}{2n}\sum_{w}^{}w^2$

Notice that, for every (or almost every?) graph in the book thus far, the value of C has stayed between 0 and 1. This makes sense, as the total cost is proportional to 1/n (where n is the total number of training data, not the number of weights), and individual costs seem to generally output a value between 0 and 1.

So, let's try substituting

$w&space;=&space;n^{k}$

for various values of k, where n is the total number of weights:

$k&space;=&space;1:&space;\sum_{w}^{}&space;w^{2}&space;=&space;n^{3},&space;C&space;=&space;C_{0}&space;+&space;\frac{\lambda&space;n^{2}}{2}$,

$k&space;=&space;0:&space;\sum_{w}^{}&space;w^{2}&space;=&space;n,&space;C&space;=&space;C_{0}&space;+&space;\frac{\lambda&space;}{2}$,

$k&space;=&space;-\frac{1}{2}:&space;\sum_{w}^{}&space;w^{2}&space;=&space;1,&space;C&space;=&space;C_{0}&space;+&space;\frac{\lambda&space;}{2n}$,

$k&space;=&space;-1:&space;\sum_{w}^{}&space;w^{2}&space;=&space;\frac{1}{n},&space;C&space;=&space;C_{0}&space;+&space;\frac{\lambda&space;}{2n^{2}}$

When k is greater than or equal to 1, we see that the Altered Cost is way, way above 1, so pretty much all of the Cost is due to the weights being too large. When k = 0, a value of 5.0 for lambda would still leave us with 2.5 of the total cost being due to large weights. Again, Cost should probably be between 0 or 1 if the network is going to be learning primarily from SDC.

k = -1/2 leaves a pretty low contribution of the Altered Cost to be due to large weights (far below 1), so we should expect the effects of adjusted cost to wean off somewhere near w = 1/√n

P.S.

1. To whoever gave the answer on the chalkboard, n and 𝑁𝑤 are meant to be the same number.

2. k = -1/4, for example, gives a higher contribution to adjusted cost that is still below the threshold of what I would call meaningfully disruptive for many values of n, especially large ones. However, for n = 4000 (which it might be for something simple), this would still lead to an Altered Cost contribution of 0.025*lambda, which is above 0.1 for lambda = 5. I think bounding the cut-off to be between k = -1/4 and -1/2 is good enough for a heuristic, and for large n, the difference between the results of k=-1/4 and k = -1/2 shouldn't be too numerically large.

1. In L2 weights are updated by: $$w' \rightarrow w(1-\frac{\eta\lambda}{n})-\frac{\eta}{m}\sum_x\frac{\partial C_x}{\partial w}$$. During the first epochs of training most of the neurons in the network are saturated due to the old weight initialization method. Thus, the second term would be negligible, as the unregularized cost $$C_x$$ won't be affected much by small changes to the weight (in the hidden layer at least). Thus, the weight decay would dominate the training.

2. If $$n>>\eta\lambda$$ then one can use the Taylor expansion $$e^x\approx 1 +x$$ and get $$e^-\frac{\eta\lambda}{n}\approx1-\frac{\eta\lambda}{n}$$.

3. The weight decay keeps dominating training as long as the neurons in the network are saturated. Saturation happens when the weighted sum of a neuron, $$z$$, is either very large or very small, that is, when $$|z|\gg1$$. Conversly, a neuron is not saturated when $$|z|\sim1$$. Recall that $$z=\sum_i w_ix_i + b$$. $$b$$ is of order unity (it was drawn from a standard normal distribution), and so does $$x_i$$ (neuron activations can get values between $$0$$ and $$1$$), so $$z\sim nw$$ where now we are using $$n$$ as the number of neurons in the layer. The number of weight, $$n_w$$, goes roughly as the number of neurons in the layer squared, so $$n\approx \sqrt{n_w}$$, so we get that $$z\sim \sqrt{n_w}w$$. Demanding that $$z$$ be of order unity, we finally get the desired result $$w\sim \frac{1}{\sqrt{n_w}}$$.