-1
$\begingroup$

HI I am using a fully connected network that uses sigmoid if we feed a a big enough weights the sigmoid function will finally become 1 or 0 , is there any solution to avoid this ?

and will this lead to classical sigmoid problems vanishing gradient or exploding gradient ?

Thank you

$\endgroup$
3
  • $\begingroup$ You can try using the ReLU activation function. $\endgroup$ – Shubham Panchal Sep 11 '19 at 11:48
  • $\begingroup$ ShubhamPanchal I need to calculate probability that is why I assume I am tied to sigmoid $\endgroup$ – ou2105 Sep 11 '19 at 12:51
  • 2
    $\begingroup$ if the function value becoming 0 or 1 due to float precision issues you could just add/subtract a small epsilon (like 1e-6), this should not hurt your results $\endgroup$ – nyro_0 Sep 11 '19 at 14:21
0
$\begingroup$

The sigmoid function is an activation function and would produce values between 0 and 1. So basically if you want values other than 0 and 1 you should not be using sigmoid. Secondly, if you are using sigmoid in the last layer of your network you can replace it by softmax function.

$\endgroup$
0
$\begingroup$

If you want the probability of different classes as the output, you should use SOFTMAX activation function.

working of a softmax

It outputs probability of all the classes based on the above formula.

$\endgroup$
0
$\begingroup$

The graph of sigmoid is shown below

enter image description here

When you have very large weights because of the activation sigmod(w^tx + b) where w and b your large weights you will end up with passing a very large positive or negative number to sigmoid functions. Very large positive number will return 1 and similarly very large negative number will return 0 by the sigmoid function (as shown in sigmoid graphs)).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.