3
$\begingroup$

How did we arrive at the sigmoid function for calculating probabilities?

Why not use some other function that " squashes " the values to lie between [0, 1]. Maybe even just normalise the values so they all add up to one.

$\endgroup$
3
  • 3
    $\begingroup$ The derivative of this function is nice, making it invaluable for gradient descent. $\endgroup$ – Emre Mar 30 '17 at 4:42
  • 4
    $\begingroup$ stats.stackexchange.com/questions/162988/… $\endgroup$ – SmallChess Mar 30 '17 at 5:09
  • $\begingroup$ This question starts with a false premise. Many people use other linking functions. Logit is the most popular, but probit is very popular as well. $\endgroup$ – Ricardo Cruz Mar 31 '17 at 9:28
3
$\begingroup$

I think a really nice explanation for the popularity of the sigmoid function is in these lecture notes (http://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf)

  1. The most obvious idea is to let $p(x)$ be a linear function of $x$. Every increment of a component of $x$ would add or subtract so much to the probability. The conceptual problem here is that $p$ must be between $0$ and $1$, and linear functions are unbounded. Moreover, in many situations we empirically see “diminishing returns” — changing $p$ by the same amount requires a bigger change in $x$ when $p$ is already large (or small) than when $p$ is close to $1/2$. Linear models can’t do this.
  2. The next most obvious idea is to let $\log p(x)$ be a linear function of $x$, so that changing an input variable multiplies the probability by a fixed amount. The problem is that logarithms are unbounded in only one direction, and linear functions are not.
  3. Finally, the easiest modification of $\log p$ which has an unbounded range is the logistic (or logit) transformation, $\log (p(1−p))$ . We can make this a linear function of $x$ without fear of nonsensical results. (Of course the results could still happen to be wrong, but they’re not guaranteed to be wrong.)
$\endgroup$
1
  • $\begingroup$ > The problem is that logarithms are unbounded in only one direction, and linear functions are not. Aren't logarithms actually unbounded in both directions? $\endgroup$ – pX0r Jun 8 '18 at 6:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.