I am new to ds and stats in general. I read numerous article to understand logistic regression. I got some idea why it works and how it fits to scattered plot of 1s and 0s when target variable is binary. however one piece of puzzle I still don't understand is how come someone derived that

ln(p/1-p) = B0 + B1X1 + ..

I see all articles assume that this is the link function and then go on talking about how it solves our regression problem for binary variable. but how this link function came about.


p is a probability so it is strictly between 0 and 1. So ln(p/(1-p)) is:

for p = 0: ln(0/1) = -Inf

for p = 1: ln(1/0) = +Inf

So now you've rescaled the probability to +/- Inf. In the GLM framework you can use practically any function that scales probability to +/-Inf (see any GLM textbook, or https://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models).

How did it come about? Well, mathematicians and statisticians realised they needed a function with the above properties, had a think, came up with a few, decided on the ones that had tractable asymptotic properties, explored how they worked with real data and decided the sensible ones were logit (as above), probit (see references) and a few others. Of course you should always test your model assumptions against your data, and choice of link is just another one of those assumptions, just like assuming linearity in a covariate.

| improve this answer | |
  • $\begingroup$ I see. 2nd answer on that link made sense. so any function whose value range +/- Inf can be fitted via GLM in general? is that an assumption here? $\endgroup$ – nir Aug 26 '16 at 15:59

One intuitive answer as to why logistic function comes up is to look at Logistic Regression from the angle of generative model, which results in Linear Discriminant Analysis model.

Basically, the idea is that instead of directly modeling the likelihood $p(y|x)$ like in logistic regression. You model the class-conditional $p(x|y)$ and $p(y)$, then derive the output $p(y)$ via Bayes' rule.

$$ p(y|x) = \frac{p(x|y) p(y)}{p(x)} $$

It turns out that if you model the input by a Gaussian distribution or any distribution in the exponential family (with the same dispersion parameter for two classes), then your likelihood is a logistic function of $x$

$$ p(y = 1 | x) = \frac{p(x|y=1)p(y=1)}{p(x|y=1)p(y=1) + p(x|y=0)p(y=0)}$$

where $$ p(x|y=1) = \mathcal{N(\mu_1, \Sigma)} $$ $$ p(x|y=1) = \mathcal{N(\mu_0, \Sigma)} $$

after some simplification

$$ p(y=1|x) = \frac{1}{1 + exp(-w^Tx - b)} $$

where $ w = \Sigma^{-1}(\mu_0 - \mu_1) $

That explains why the logistic function turns up

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.