8
$\begingroup$

sigmoid function could be used as activation function in machine learning.

$${\displaystyle S(x)={\frac {1}{1+e^{-x}}}={\frac {e^{x}}{e^{x}+1}}.}$$

If substitute e with 2, 

def sigmoid2(z):
    return 1/(1+2**(-z))
x = np.arange(-9,9,dtype=float)
y = sigmoid2(x)
plt.scatter(x,y)

the plot looks similar.

enter image description here

Why does the logistic function use $e$ rather than 2?

Improve this question
$\endgroup$
0

3 Answers 3

Reset to default
11
$\begingroup$

Since you are going to minimize later on the log likelihood, there is actually no big difference between $\log 2^x=x * \log2$ and $\log e^x=x$. You see the difference is simply a constant.
Nevertheless one could argue to use $2^x$ instead of $e^x$ und also use $\log_2$ instead of $\log$ when it comes to the optimizing step. In fact it is possible to use $2^x$ and also many other functions, which show some desired properties. Which are:

  • $\lim\limits_{x \rightarrow \infty}{f(x)}=1$
  • $\lim\limits_{x \rightarrow -\infty}{f(x)}=0$
  • $f(x) = -f(-x) + 1$, (symmetric in $(0, 0.5)$

Here is an example of suitable functions from wikipedia.

Improve this answer
$\endgroup$
4
  • 9
    $\begingroup$ I think it's also worth pointing out that one nice reason to use $e$ as the base is that the derivative of $\sigma(x)=\frac{1}{1+e^{-x}}$ is $\sigma'(x)=\sigma(x)(1-\sigma(x))$. Without doing the actual computation, I think if the base was different the formula would only differ by a constant again, but it's a nice property that is specific to $e$. $\endgroup$
    – Calvin Godfrey
    Jun 6, 2019 at 16:55
  • $\begingroup$ Same goes for $2^x$ when using $\log_2$. $\endgroup$
    – Andreas Look
    Jun 6, 2019 at 20:44
  • $\begingroup$ @AndreasLook I'm not sure what you mean. If you use $2^{-x}$ then there's an extra factor of $\ln(2)$ in the derivative (like Calvin Godfrey said). $\endgroup$
    – sfmiller940
    Jun 12, 2019 at 20:08
  • $\begingroup$ No, check out binary logarithm. $\log_2 (2^x)=x$. $\endgroup$
    – Andreas Look
    Jun 13, 2019 at 19:32
4
$\begingroup$

So there are many functions that look sigmoid including the 2 you mentioned, but there are reasons why $e$ is special. The main reason it that the logistic function was originally used to model population growth. And populations, much like interest, can compound over time. Thus, the $e$ becomes a very natural object for this reason. In addition, for theoretical reasons concerning the canonical link function of a glm the logistic is one of the theoretically simplest objects to work with which makes it easy to prove things with.

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ thanks for your answer. what does "canonical link function of a glm" mean? $\endgroup$
    – JJJohn
    Jun 6, 2019 at 9:22
  • $\begingroup$ @baojieqh For all generalized linear models, one needs to specify a member of the exponential family of distributions. These distributions all share a property where they can be written in such a way so that a function of the scale parameter of the distribution sits "by itself" in an exponent (and the function is only a function of the scale parameter). This function is what people refer to as the canonical link function. For the bernoulli/binomial distribution, where the scale parameter is p, it turns out that this function is ln(p/(1-p)) which is the logit link function. $\endgroup$
    – aranglol
    Jun 6, 2019 at 23:55
  • $\begingroup$ Hence, the canonical link function for the logistic regression, which assumes a Bernoulli distribution for each row, is the logit link. There are other more theoretical properties as well that make the canonical link function desirable. But it is technically not necessary to use it, you could use the probit for example. $\endgroup$
    – aranglol
    Jun 6, 2019 at 23:58
  • $\begingroup$ @aranglol thanks for you comments, would you please take a look at this link math.stackexchange.com/q/3253634/656371 $\endgroup$
    – JJJohn
    Jun 7, 2019 at 0:37
  • $\begingroup$ This seems to be just a hand-waving appeal to the claim that "$e$ is special", without giving any justification about why $e$ is special. Really, the only specialness is the convenience that $\tfrac{d}{dx}a^x=a^x\ln a$, which means that $\tfrac{d}{dx}e^x=e^x$. $\endgroup$
    – David Richerby
    Jun 7, 2019 at 9:21
0
$\begingroup$

It comes from the basic assumption of the model that there exists a continuous/latent/unobservable $Y^*$ that relates somehow to the observed values of $Y$. The model further assumes that $Y=1$ if the signal of $Y^*$ is above some threshold, and otherwise $Y=0$. The third and last assumption is that the underlying distribution of $Y*$ is the logistic distribution. Once you have these assumptions, it is only a matter of algebra to derive the model.

You can read more details at my blog.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.