I'm working through http://neuralnetworksanddeeplearning.com/chap3.html, textbook by Michael Nielsen.

While most problems are rather straightforward, I'm very stuck on these two questions:

Sketch a heuristic argument that: (1) supposing $\lambda$ is not too small, the first epochs of training will be dominated almost entirely by weight decay; (2) provided $\eta\lambda \ll n$ the weights will decay by a factor of $ \exp(−\eta\lambda/m)$ per epoch;

To give a self-sufficient context, the weight decay function is:

$w \rightarrow \left(1-\frac{\eta \lambda}{n}\right) w -\frac{\eta}{m} \sum_x \frac{\partial C_x}{\partial w}$,

and cost function is

$C = -\frac{1}{n} \sum_{xj} \left[ y_j \ln a^L_j+(1-y_j) \ln (1-a^L_j)\right] + \frac{\lambda}{2n} \sum_w w^2$

(Classic logloss)

Weights in the network are initiated as $N(0,1)$.

My intuition for (1) goes as follows: As weight initialisation is non-normalized by size of previous layer, activation values are likely to be either $1$ or $0$ with equal probability (which is pointed out by the author). $y_j$ has $0.1$ probability of being 1. Given that $\frac{\partial C_x}{\partial w} = a^{L-1}_k(a^L_j - y_j)$ for logloss and sigmoid neurons, expected value of $\frac{\partial C_x}{\partial w}$ on the first iteration is $0.2$. Given that $\lambda$ is arbitrary, we can be sure that $|\frac{\eta \lambda}{n} w|$ can be larger than $|\frac{\eta}{m} \sum_x \frac{\partial C_x}{\partial w}|$

Despite questions being rather informal, I don't think this solution is a good one.

As for the (2), I lack any ideas whatsoever, except driving $\frac{\eta \lambda}{n}$ to zero, but that seems to have zero chance of yielding the required result.

I hope someone can give me a hand in this. Thank you!

P.S. There is a chance I'm going to the wrong overflow, but I used https://meta.stackexchange.com/questions/130524/which-stack-exchange-website-for-machine-learning-and-computational-algorithms question as a guidance for overflow choice.

A note: I had asked similiar question here, but that might not be the best overflow for this question. I'll put a link to whichever get's answered first :)

  • 1
    $\begingroup$ Hint: $\lim_{n\to \infty} \left(1-\eta \lambda/n \right)^{n/m} = \exp(-\eta \lambda/m)$ $\endgroup$
    – Emre
    Nov 10, 2016 at 5:31
  • $\begingroup$ @Emre Thanks! The problem is however getting $n/m$ thing to the power. I don't see how to get it from $\frac{\eta}{m} \sum_x \frac{\partial C_x}{\partial w}$ up there. There is no reason to have $n$ there - we are working with $m$ observations, and normalize the result by $m$. $\endgroup$
    – Loiisso
    Nov 10, 2016 at 14:17
  • $\begingroup$ Aha, wait. $n/m$ is a number of updates per epoch. So if (1) is correct (which means $\frac{\eta}{m} \sum_x \frac{\partial C_x}{\partial w}$ does nothing, than we are left with $(1-\eta \lambda/n)$. So if $lim_{n\to \infty}$, than first epoch shall yield the statement in your comment. $\endgroup$
    – Loiisso
    Nov 10, 2016 at 15:45

1 Answer 1


Ok, I have found the source of confusion, thanks to @Emre for the insight. The task talks about epochs of training, while I was trying to solve one batch update. I think this confusion is statistically possible enough to validate existence of this question :)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.