# Speed decay proof for L2 regularization and non-normalizied weight initiation

I'm working through http://neuralnetworksanddeeplearning.com/chap3.html, textbook by Michael Nielsen.

While most problems are rather straightforward, I'm very stuck on these two questions:

Sketch a heuristic argument that: (1) supposing $\lambda$ is not too small, the first epochs of training will be dominated almost entirely by weight decay; (2) provided $\eta\lambda \ll n$ the weights will decay by a factor of $\exp(−\eta\lambda/m)$ per epoch;

To give a self-sufficient context, the weight decay function is:

$w \rightarrow \left(1-\frac{\eta \lambda}{n}\right) w -\frac{\eta}{m} \sum_x \frac{\partial C_x}{\partial w}$,

and cost function is

$C = -\frac{1}{n} \sum_{xj} \left[ y_j \ln a^L_j+(1-y_j) \ln (1-a^L_j)\right] + \frac{\lambda}{2n} \sum_w w^2$

(Classic logloss)

Weights in the network are initiated as $N(0,1)$.

My intuition for (1) goes as follows: As weight initialisation is non-normalized by size of previous layer, activation values are likely to be either $1$ or $0$ with equal probability (which is pointed out by the author). $y_j$ has $0.1$ probability of being 1. Given that $\frac{\partial C_x}{\partial w} = a^{L-1}_k(a^L_j - y_j)$ for logloss and sigmoid neurons, expected value of $\frac{\partial C_x}{\partial w}$ on the first iteration is $0.2$. Given that $\lambda$ is arbitrary, we can be sure that $|\frac{\eta \lambda}{n} w|$ can be larger than $|\frac{\eta}{m} \sum_x \frac{\partial C_x}{\partial w}|$

Despite questions being rather informal, I don't think this solution is a good one.

As for the (2), I lack any ideas whatsoever, except driving $\frac{\eta \lambda}{n}$ to zero, but that seems to have zero chance of yielding the required result.

I hope someone can give me a hand in this. Thank you!

P.S. There is a chance I'm going to the wrong overflow, but I used https://meta.stackexchange.com/questions/130524/which-stack-exchange-website-for-machine-learning-and-computational-algorithms question as a guidance for overflow choice.

A note: I had asked similiar question here, but that might not be the best overflow for this question. I'll put a link to whichever get's answered first :)

• Hint: $\lim_{n\to \infty} \left(1-\eta \lambda/n \right)^{n/m} = \exp(-\eta \lambda/m)$ – Emre Nov 10 '16 at 5:31
• @Emre Thanks! The problem is however getting $n/m$ thing to the power. I don't see how to get it from $\frac{\eta}{m} \sum_x \frac{\partial C_x}{\partial w}$ up there. There is no reason to have $n$ there - we are working with $m$ observations, and normalize the result by $m$. – Loiisso Nov 10 '16 at 14:17
• Aha, wait. $n/m$ is a number of updates per epoch. So if (1) is correct (which means $\frac{\eta}{m} \sum_x \frac{\partial C_x}{\partial w}$ does nothing, than we are left with $(1-\eta \lambda/n)$. So if $lim_{n\to \infty}$, than first epoch shall yield the statement in your comment. – Loiisso Nov 10 '16 at 15:45