I have been reading through this book and am trying to do the exercises.
The problem is "Connecting regularization and the improved method of weight initialization" part 3. We have to use a heuristic argument to prove "the weight decay will tail off when the weights are down to a size around $1/\sqrt{n},$ where $n$ is the total number of weights in the network."
For more context on the same problem (Parts 1 and 2):
Heuristic argument for Weight decay and regularization
Speed decay proof for L2 regularization and non-normalizied weight initiation
The relevant equation appears to be this:
$$C= -\frac{1}{n_t} \sum_{xj} [y_j \ln{a^L_j}+(1−y_j)\ln{(1−a^L_j)}]+\frac{\lambda}{2n_t}\sum_{w}w^2.$$
(The book seems to use the $n$ notation to represent two different things. I've changed it so that $n_t$ refers to the size of the training set, and $n_w$ refers to the number of weights in the network)
I think it has something to do with the last term, $\frac{\lambda}{2n}\sum_{w}w^2$, because if we substitute $w=\frac{1}{\sqrt{n_w}}$ then the whole term simplifies to $\frac{\lambda}{2n}$. This would mean that the partial derivative with respect to the weight becomes $\frac{\partial C}{\partial w}$ instead of $\frac{\partial C}{\partial w}+\frac{\lambda}{n}w$. Is this enough of an explanation to conclude that weight decay falls off once the weights are around $\frac{1}{\sqrt{n}}?$