Suppose we have the following fully connected network made of perceptrons with a sign function as the activation unit, what issue arises, when trying to train this network with gradient descent?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ What do you mean with "what issue arises"? Have you encountered issues when training such a network already? $\endgroup$– OxbowerceMar 13, 2021 at 21:22
-
$\begingroup$ @Oxbowerce This is an exercise from a past exam, which suggests that there is an issue. I have not used a network of this type for an actual tasks/ encountered an issue myself. $\endgroup$– WhatAMeshMar 13, 2021 at 21:26
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
what issue arises, when trying to train this network with gradient descent?
The activation function is sign function or signum function (A little modified).
So, its Derivative will be 0 at all the points
Hence, the Gradient descent won’t be able to make progress in updating the weights and backpropagation will fail.
