I'm working through http://neuralnetworksanddeeplearning.com/chap3.html, textbook by Michael Nielsen.
While most problems are rather straightforward, I'm very stuck on these two questions:
Sketch a heuristic argument that: (1) supposing $\lambda$ is not too small, the first epochs of training will be dominated almost entirely by weight decay; (2) provided $\eta\lambda \ll n$ the weights will decay by a factor of $ \exp(−\eta\lambda/m)$ per epoch;
To give a self-sufficient context, the weight decay function is:
$w \rightarrow \left(1-\frac{\eta \lambda}{n}\right) w -\frac{\eta}{m} \sum_x \frac{\partial C_x}{\partial w}$,
and cost function is
$C = -\frac{1}{n} \sum_{xj} \left[ y_j \ln a^L_j+(1-y_j) \ln (1-a^L_j)\right] + \frac{\lambda}{2n} \sum_w w^2$
(Classic logloss)
Weights in the network are initiated as $N(0,1)$.
My intuition for (1) goes as follows: As weight initialisation is non-normalized by size of previous layer, activation values are likely to be either $1$ or $0$ with equal probability (which is pointed out by the author). $y_j$ has $0.1$ probability of being 1. Given that $\frac{\partial C_x}{\partial w} = a^{L-1}_k(a^L_j - y_j)$ for logloss and sigmoid neurons, expected value of $\frac{\partial C_x}{\partial w}$ on the first iteration is $0.2$. Given that $\lambda$ is arbitrary, we can be sure that $|\frac{\eta \lambda}{n} w|$ can be larger than $|\frac{\eta}{m} \sum_x \frac{\partial C_x}{\partial w}|$
Despite questions being rather informal, I don't think this solution is a good one.
As for the (2), I lack any ideas whatsoever, except driving $\frac{\eta \lambda}{n}$ to zero, but that seems to have zero chance of yielding the required result.
I hope someone can give me a hand in this. Thank you!
P.S. There is a chance I'm going to the wrong overflow, but I used https://meta.stackexchange.com/questions/130524/which-stack-exchange-website-for-machine-learning-and-computational-algorithms question as a guidance for overflow choice.
A note: I had asked similiar question here, but that might not be the best overflow for this question. I'll put a link to whichever get's answered first :)
