Help in understanding the maths behind Logistic Regression

Question

I am following the lecture notes available https://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf

I cannot understand how Eqs 12.4 and 12.5 come,

• why the Bernoulli probability has $1-p(x)$ in the denominator,
• how come $p(x) = \exp(\beta + \beta^Tx)$
• and how $log \frac{p(x)}{1-p(x)}$ evaluates to $\beta + \beta^Tx$.

In general $\beta$ is the parameter of the model but I don't quite follow how come the log expression evaluates to it. Is there some mathematical formula which is skipped that is used to evaluate the log expression? This is crucial for me to know as these values are substituted in eq 12.10 where $p(x) = \exp(\beta + \beta^Tx)$

Answer 1

This feels like a bit of a convoluted way to introduce the concept, but alright :D Let me start at a slightly different point.

1. Maybe in Machine Learning or in other places you have encountered the $sigmoid$ function: $$ sigmoid(S) = \frac{e^S}{1+e^S} = \frac{1}{1+e^{-S}} $$ The sigmoid has the nice property to map any real number $S$ to a number between 0 and 1. This is super when dealing with models that have to represent probabilities.

2. As in the slides they are looking for a (conditional) probability $p(x)$ they go ahead and implicitly set $$ p(S) = sigmoid(S) $$ They just do it around an extra corner. In order to be consistent with the slides (and to save some tedious writing) I'll now keep talking about the probability $p(x)$ and think of the $sigmoid$-function.

3. $p(S)$ (i.e. the $sigmoid$) has an inverse function, called the $logit$. Let's try to find the inverse of $p(S)$: $$ p = \frac{1}{1+e^{-S}} $$ and from there we move some stuff around $$ 1+e^{-S} = \frac{1}{p} $$ $$ e^{-S} = \frac{1}{p} - 1 $$ $$ -S = \log \left[\frac{1}{p} - 1 \right] $$ $$ S = \log \left[\frac{1}{\frac{1}{p} - 1} \right] $$ $$ S = \log \left[\frac{p}{1 - p} \right] $$ This is where they start in your notes. We can take any number $S$, punch it into the logit, solve for $p$ and Boom! we've got ourselves a nice conditional probability.

4. So the next question is: what is $S$? $S$ is supposed to be some function of $x$. There is a wide variety of functions you could use. You could even put a massive Neural Network, but for now, let's stick to linear regression $$ S(x) = \beta_0 + \beta x $$ If you decide to go for $S(x)$ being linear, you can now go step 3. backwards and end up at step 1. with the expression they also show in the notes $$ p(x) = \frac{e^{\beta_0 + \beta x}}{1+e^{\beta_0 + \beta x}} = \frac{1}{1+e^{-(\beta_0 + \beta x)}} $$

So to answer your questions:

$1-p(x)$ is not in the denominator of the distribution, it seems to me they just introduce it in the $\log\frac{p}{1-p}$ in order to end up with the $sigmoid$.

$p(x)$ is not equal to $e^{\beta_0 + \beta x}$, but equal to the $sigmoid$.

$\log \frac{p}{1-p}$ is declared to be equal to $\beta_0 + \beta x$ in order to end up with the $sigmoid$.

I think it would have been more straight-forward to say that you want to map a linear function to something between 0 and 1, which you can do with the sigmoid and then you would have been done in one step. It seems a bit weird to introduce the inverse of the sigmoid, claiming that this was some property you want and then solve for $p$, but that might be a matter of taste.