Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
HI I am using a fully connected network that uses sigmoid if we feed a a big enough weights the sigmoid function will finally become 1 or 0 , is there any solution to avoid this ?
and will this lead to classical sigmoid problems vanishing gradient or exploding gradient ?
Thank you
The sigmoid function is an activation function and would produce values between 0 and 1. So basically if you want values other than 0 and 1 you should not be using sigmoid. Secondly, if you are using sigmoid in the last layer of your network you can replace it by softmax function.
If you want the probability of different classes as the output, you should use SOFTMAX activation function.
It outputs probability of all the classes based on the above formula.
The graph of sigmoid is shown below
When you have very large weights because of the activation sigmod(w^tx + b) where w and b your large weights you will end up with passing a very large positive or negative number to sigmoid functions. Very large positive number will return 1 and similarly very large negative number will return 0 by the sigmoid function (as shown in sigmoid graphs)).
