3
$\begingroup$

I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that any gradients flowing back from a neuron following a sigmoid will all share the same sign as the upstream gradient flowing into that neuron. She then says that a consequence of these weight updates sharing the same sign is a sub-optimal zigzag gradient descent path.

I understand this phenomenon when zoomed in on a single neuron. However, since upstream gradients flowing into a layer can be of different signs, it's still possible to get a healthy mixture of positive and negative weight updates in a layer. Therefore, I'm having trouble understanding how using sigmoid results in this zigzag descent path, except for in the case where the upstream gradients are all of the same sign (which intuitively seems uncommon). It seems to me that if this suboptimal descent is important enough to be highlighted in the lecture, that it must be more common than that.

I'm wondering if the issue is "reduced entropy" among the weight updates, rather than all weight updates in the network sharing the same sign. That is, zigzagging in a subset of the dimensions. For example, say a network using sigmoid has four weights in a layer with two neurons: w1, w2, w3, and w4. The updates to w1 and w2 could be positive, while the updates to w3 and w4 could be negative if the two upstream gradients differ in sign. However, it wouldn't be possible for w1 and w3 to be positive, and w2 and w4 to be negative. Is this the limitation of sigmoid that the Stanford lecture is referring to, assuming the second combination of weight updates was the optimal one?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Yes, the lecturer is refering to a single neuron. It's true that in the same layer we could have different signs of updates. However, by using sigmoid function, all the weights connecting to a single neuron will be updated increasing its value or decreasing it (and not both at the same time) $\rightarrow$ zig-zagging in order to reach the optimal value of the weights. $\endgroup$
    – Javier TG
    Oct 31, 2020 at 8:26
  • 1
    $\begingroup$ Thanks Javier. To make sure I understand your response, are you saying it’s correct that with sigmoid we zigzag in a subset of dimensions? That is, constraining the gradients flowing back from a single node to share the same sign is enough to cause the behavior, regardless of what the rest of the network is doing? $\endgroup$
    – Churchjm
    Oct 31, 2020 at 15:57
  • 1
    $\begingroup$ Yes, I think we are on the same page. Concretely, this happens because the update of a weight that connects a neuron $j$ with a neuron $k$ is given by a quantity proportional to: $$\frac{\partial C}{\partial w^l_{kj}}= \delta^l_k \,\,a_j^{l-1}$$ Where $C$ is the cost function, $a_j^{l-1}$ the activation of the neuron $j$ and $\delta^l_k$ the "error" term for the neuron $k$. $\delta^l_k$ is just a scalar, hence, if all $a_j^{l-1}$ are positive (this happens with sigmoid), then the updates of all the weights that connect to a neuron $k$ will have the same sign $\endgroup$
    – Javier TG
    Oct 31, 2020 at 16:12
  • 1
    $\begingroup$ Thanks for your comments Javier. It seems I can't mark a comment as an answer though. If you'd like to re-post this as an answer, I'll accept it. $\endgroup$
    – Churchjm
    Nov 1, 2020 at 18:01

1 Answer 1

Reset to default
1
$\begingroup$

Two primary reasons sigmoid is a sub-optimal activation function for gradient descent:

  1. A node's activation saturates at either tail of 0 or 1. The resulting gradient is very small at those points, thus a very small learning signal.
  2. The maximum gradient is .25. Since this value is less than 1, the error signal is attenuated and can result in vanishing gradients after many layers of backpropagation.
Improve this answer
$\endgroup$
2
  • $\begingroup$ It might be useful to add that during backprop, the gradient is typically multiplied by the derivative of the activation function. $\endgroup$
    – Mr Tsjolder from codidact
    Jun 22 at 8:51
  • $\begingroup$ Also, a logistic sigmoid at the end of the network (in combination with binary cross-entropy) is not that much of a problem. (see e.g. this answer for some intuition) $\endgroup$
    – Mr Tsjolder from codidact
    Jun 22 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.