Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am new to ds and stats in general. I read numerous article to understand logistic regression. I got some idea why it works and how it fits to scattered plot of 1s and 0s when target variable is binary. however one piece of puzzle I still don't understand is how come someone derived that
ln(p/1-p) = B0 + B1X1 + ..
I see all articles assume that this is the link function and then go on talking about how it solves our regression problem for binary variable. but how this link function came about.
p is a probability so it is strictly between 0 and 1. So ln(p/(1-p)) is:
for p = 0: ln(0/1) = -Inf
for p = 1: ln(1/0) = +Inf
So now you've rescaled the probability to +/- Inf. In the GLM framework you can use practically any function that scales probability to +/-Inf (see any GLM textbook, or https://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models).
How did it come about? Well, mathematicians and statisticians realised they needed a function with the above properties, had a think, came up with a few, decided on the ones that had tractable asymptotic properties, explored how they worked with real data and decided the sensible ones were logit (as above), probit (see references) and a few others. Of course you should always test your model assumptions against your data, and choice of link is just another one of those assumptions, just like assuming linearity in a covariate.
One intuitive answer as to why logistic function comes up is to look at Logistic Regression from the angle of generative model, which results in Linear Discriminant Analysis model.
Basically, the idea is that instead of directly modeling the likelihood $p(y|x)$ like in logistic regression. You model the class-conditional $p(x|y)$ and $p(y)$, then derive the output $p(y)$ via Bayes' rule.
$$ p(y|x) = \frac{p(x|y) p(y)}{p(x)} $$
It turns out that if you model the input by a Gaussian distribution or any distribution in the exponential family (with the same dispersion parameter for two classes), then your likelihood is a logistic function of $x$
$$ p(y = 1 | x) = \frac{p(x|y=1)p(y=1)}{p(x|y=1)p(y=1) + p(x|y=0)p(y=0)}$$
where $$ p(x|y=1) = \mathcal{N(\mu_1, \Sigma)} $$ $$ p(x|y=1) = \mathcal{N(\mu_0, \Sigma)} $$
after some simplification
$$ p(y=1|x) = \frac{1}{1 + exp(-w^Tx - b)} $$
where $ w = \Sigma^{-1}(\mu_0 - \mu_1) $
That explains why the logistic function turns up
