Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I was asking myself why eulers number was used in the sigmoid function 1/(1+e^-x) instead of any other constant like for example 2 or 3?
I am pretty new to data science stuff, but I read somerwhere that eulers number is the natural growth of a curve, so would this mean, that eulers number is used in the sigmoid function because it makes it possible to output values that are evenly distributed between the values 0 and 1?
Euler's number pops up in a lot of places naturally; not quite something to do with growth rates but arises easily in common limits.
The form of the sigmoid function wasn't chosen because its derivative has a nice property, although that's true. It also wasn't chosen because it's a function with range (0,1); many functions do that.
The sigmoid function arises because it's the correct answer to a common type of problem. We think of logistic regression as a common classifier predicting a class probability, but it really is regression under the hood. It's not regressing the probability though, but log-odds of the probability (logit function). This is the right way to apply regression, given assumptions about the distribution of errors in a classification problem, which aren't the same as in a simple linear regression.
The sigmoid function is the inverse of the logit link function. That's why it's there. It gets from the regression output to the actual desired output, a probability. The logit function is there because it is implied by the assumption about the distribution of the 0/1 dependent variable.
That's actually it. It has to be there in certain types of common problems because it's the answer to those problems, not because it was chosen for nice properties.
Because you need to minimize the error which contains output i.e. you take derivative of error and set it to zero (basically). If output is coming out of the sigmoid then you have a very nice property: The derivative of sigmoid can be written using itself!
$$\sigma (x) = \frac{1}{1+e^{-x}}.$$ $$\frac{d\sigma (x)}{d(x)} = \sigma (x)\cdot (1-\sigma(x)).$$
If you use a constant $a$, then a term $ln(a)$ will destroy the beautiful property!
PS: The constant just scales the shape of function i.e. if you increase it, you reach values close to 0 or 1 earlier. See bellow:
If I understand your question correctly, it's why do we try to fit $\frac{1}{1+e^{-a(x-b)}}$ to our data rather than $\frac{1}{1+\kappa^{-a(x-b)}}$ ? (I've used the univariate case here just to keep things simple)
Consider, that for any (positive) $\kappa$, we can write it as $\kappa = e^{c}$, and thus $\frac{1}{1+\kappa ^{-a(x-b)}}=\frac{1}{1+(e^{c})^{a(x-b)}}=\frac{1}{1+e^{ac(x-b)}}$
So from a solution perspective, it doesn't matter whether you use the natural exponent, or any other, the curve in this function of families which maximises your likelihood will be the same, and your parameter (in the above example, a) will just take a different value, depending on which exponent you use.
This is true for any positive $\kappa$, and for negative $\kappa$, you lose the basic property that your output probability must be between 0 and 1.
So in conclusion, you don't have to use the natural exponent, you can use any positive one, but using the natural exponent keeps gradient calculations simple, and most likely any other follow on calculation you might want to do. You could do it with other exponents, but then you'd have to keep track of trailing logarithms.
They are equivalent. For example, $0.5^{x\cdot\beta}=e^{(\ln0.5)\cdot x \cdot \beta}=e^{x \cdot \beta’}$
using $e$ makes the derivative much more elegant.
