11
$\begingroup$

In cs231n course , it is mentioned that

If the initial weights are too large then most neurons would become saturated and the network will barely learn.

How do the neurons get saturated? Large weights may lead to a z (output of saturation) which is not cery close 0 or 1 and so doesn't let z*(1-z) to saturate

Improve this question
$\endgroup$

1 Answer 1

Reset to default
12
$\begingroup$

The sigmoid function $$ \theta(z) = \frac{1}{1+e^{-z}}$$ looks like this :

enter image description here

where $$z=w_i a_i + bias$$ for activations $a_i$ from the previous layer, and weights $w_i$ of the current neuron.

When the weights $w_i$ are too large (positive or negative), $z$ tends to be large as well, driving the output of the sigmoid to the far left (value 0) or far right (value 1). These are saturation regions where the gradient/derivative is too small, slowing down learning.

Learning slows down when the gradient is small, because the weight upgrade of the network at each iteration is directly proportional to the gradient magnitude.

Improve this answer
$\endgroup$
8
  • $\begingroup$ "When the weights wi are too large (positive or negative), z tends to be large as well". Even if the weights are large isn't there a fair chance where z could be small(because of ai)? for instance if w1=1000 and w2=1000 and if a1=1 and a2=-0.99 then z=10 $\endgroup$
    – MysticForce
    Feb 12, 2017 at 11:17
  • $\begingroup$ I understand the problem here now . For the layers deep down in a neural network , ai's are positive because sigmoid gives a positive output from previous layer.This forces z to be large. So this problem is not solely about the weights but also the positive output of sigmoid.Thanks for the insight $\endgroup$
    – MysticForce
    Feb 12, 2017 at 11:21
  • $\begingroup$ As a continuation of discussion, we could also solve this by changing the mean of sigmoid to zero by z-0.5 as output of neuron. Why isn;t this done usually? $\endgroup$
    – MysticForce
    Feb 12, 2017 at 11:22
  • $\begingroup$ It is solely about the weights. The problem is not about the sigmoid mean, it's about the sigmoid limits at inf and -inf. It's true that outputs of the sigmoid are always positive, making it spit +ve and -ve outputs does not help : Still, a very large weight (+ve or -ve) multipled by an activation (+ve or -ve, small or large) will produce a large number (+ve or -ve).....this goes through the sigmoid producing an output in a "flat" area with a tiny gradient. $\endgroup$
    – MohamedEzz
    Feb 12, 2017 at 13:50
  • 1
    $\begingroup$ A variable "Saturating" means it's slowly approaching its maximum (or minimum) possible value. From the plot above, you see sigmoid has min=0 and max=1. It comes close to "saturation" when z is below -5 or above 5. The issue is that the "saturation" behavior (i.e., slow increase/decrease) means that the gradient is small. Note that gradient is "the rate of change". Saying "the rate of change is slow" is equivalent to saying "small gradient value" $\endgroup$
    – MohamedEzz
    Feb 11, 2020 at 5:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.