In cs231n course , it is mentioned that

If the initial weights are too large then most neurons would become saturated and the network will barely learn.

How do the neurons get saturated? Large weights may lead to a z (output of saturation) which is not cery close 0 or 1 and so doesn't let z*(1-z) to saturate


The sigmoid function $$ \theta(z) = \frac{1}{1+e^{-z}}$$ looks like this :

enter image description here

where $$z=w_i a_i + bias$$ for activations $a_i$ from the previous layer, and weights $w_i$ of the current neuron.

When the weights $w_i$ are too large (positive or negative), $z$ tends to be large as well, driving the output of the sigmoid to the far left (value 0) or far right (value 1). These are saturation regions where the gradient/derivative is too small, slowing down learning.

Learning slows down when the gradient is small, because the weight upgrade of the network at each iteration is directly proportional to the gradient magnitude.

| improve this answer | |
  • $\begingroup$ "When the weights wi are too large (positive or negative), z tends to be large as well". Even if the weights are large isn't there a fair chance where z could be small(because of ai)? for instance if w1=1000 and w2=1000 and if a1=1 and a2=-0.99 then z=10 $\endgroup$ – MysticForce Feb 12 '17 at 11:17
  • $\begingroup$ I understand the problem here now . For the layers deep down in a neural network , ai's are positive because sigmoid gives a positive output from previous layer.This forces z to be large. So this problem is not solely about the weights but also the positive output of sigmoid.Thanks for the insight $\endgroup$ – MysticForce Feb 12 '17 at 11:21
  • $\begingroup$ As a continuation of discussion, we could also solve this by changing the mean of sigmoid to zero by z-0.5 as output of neuron. Why isn;t this done usually? $\endgroup$ – MysticForce Feb 12 '17 at 11:22
  • $\begingroup$ It is solely about the weights. The problem is not about the sigmoid mean, it's about the sigmoid limits at inf and -inf. It's true that outputs of the sigmoid are always positive, making it spit +ve and -ve outputs does not help : Still, a very large weight (+ve or -ve) multipled by an activation (+ve or -ve, small or large) will produce a large number (+ve or -ve).....this goes through the sigmoid producing an output in a "flat" area with a tiny gradient. $\endgroup$ – MohamedEzz Feb 12 '17 at 13:50
  • $\begingroup$ What does saturation mean in this context? I always see people explaining this phenomenon using the word saturate itself. Even when the question at hand is asking what people mean by saturate. $\endgroup$ – Nick Corona Feb 9 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.