Weight decay in neural network

Question

I have been reading through this book and am trying to do the exercises.

The problem is "Connecting regularization and the improved method of weight initialization" part 3. We have to use a heuristic argument to prove "the weight decay will tail off when the weights are down to a size around $1/\sqrt{n},$ where $n$ is the total number of weights in the network."

For more context on the same problem (Parts 1 and 2):

Heuristic argument for Weight decay and regularization

Speed decay proof for L2 regularization and non-normalizied weight initiation

The relevant equation appears to be this:

$$C= -\frac{1}{n_t} \sum_{xj} [y_j \ln{a^L_j}+(1−y_j)\ln{(1−a^L_j)}]+\frac{\lambda}{2n_t}\sum_{w}w^2.$$

(The book seems to use the $n$ notation to represent two different things. I've changed it so that $n_t$ refers to the size of the training set, and $n_w$ refers to the number of weights in the network)

I think it has something to do with the last term, $\frac{\lambda}{2n}\sum_{w}w^2$, because if we substitute $w=\frac{1}{\sqrt{n_w}}$ then the whole term simplifies to $\frac{\lambda}{2n}$. This would mean that the partial derivative with respect to the weight becomes $\frac{\partial C}{\partial w}$ instead of $\frac{\partial C}{\partial w}+\frac{\lambda}{n}w$. Is this enough of an explanation to conclude that weight decay falls off once the weights are around $\frac{1}{\sqrt{n}}?$

Answer 1

According to the book, the problem of initializing weights with too big of a standard deviation is that it is very likely to cause neurons to saturate.

But with L2 regularization, when saturation occurred only the L2 term will affect the gradient, and cause weight decay.

And when weights get small enough not to cause saturation (for example around $1 / \sqrt n$), the other term comes to affect the gradeint.

So the relative influence of L2 term decreases.

(And of course, the absolute effect of L2 term will decrease by decaying the weight.)

Why $1 / \sqrt n$?

If all of the $n$ input neurons are 1 and the standard deviations of weights are $\sigma$, the standard deviation of the input to hidden neurons will be $\sqrt{n}\sigma$.

If you want $\sqrt{n}\sigma$ to be 1 to avoid saturation, $\sigma$ should be $1 / \sqrt n$.