Skip to main content
deleted 4 characters in body
Source Link
matthiaw91
  • 1.6k
  • 6
  • 17

This feels like a bit of a convoluted way to introduce the concept, but alright :D Let me start at a slightly different point.

  1. Maybe in Machine Learning or in other places you have encountered the $sigmoid$ function: $$ sigmoid(S) = \frac{e^S}{1+e^S} = \frac{1}{1+e^{-S}} $$ The sigmoid has the nice property to map any real number $S$ to a number between 0 and 1. This is super when dealing with models that have to represent probabilities.

  2. As in the slides they are looking for a (conditional) probability $p(x)$ they go ahead and implicitly set $$ p(S) = sigmoid(S) $$ They just do it around an extra corner. In order to be consistent with the slides (and to save some tedious writing) I'll now keep talking about the probability $p(x)$ and think of the $sigmoid$-function.

  3. $p(S)$ (i.e. the $sigmoid$) has an inverse function, called the $logit$. Let's try to find the inverse of $p(S)$: $$ p = \frac{1}{1+e^{-S}} $$ and from there we move some stuff around $$ 1+e^{-S} = \frac{1}{p} $$ $$ e^{-S} = \frac{1}{p} - 1 $$ $$ -S = \log \left[\frac{1}{p} - 1 \right] $$ $$ S = \log \left[\frac{1}{\frac{1}{p} - 1} \right] $$ $$ S = \log \left[\frac{p}{1 - p} \right] $$ This is where they start in your notes. We can take any number $S$, punch it into the logit, solve for $p$ and Boom! we've got ourselves a nice conditional probability.

  4. So the next question is: what is $S$? $S$ is supposed to be some function of $x$. There is a wide variety of functions you could use. You could even put a massive Neural Network, but for now, let's stick to linear regression $$ S(x) = \beta_0 + \beta x $$ If you decide to go for $S(x)$ being linear, you can now go step 3. backwards and end up at step 1. with the expression they also show in the notes $$ p(x) = \frac{e^{\beta_0 + \beta x}}{1+e^{\beta_0 + \beta x}} = \frac{1}{1+e^{-(\beta_0 + \beta x)}} $$

So to answer your questions:

$1-p(x)$ is not in the denominator of the distribution, it seems to me they just introduce it in the $\log\frac{p}{1-p}$ in order to end up with the $sigmoid$.

$p(x)$ is not equal to $e^{\beta_0 + \beta x}$, but equal to the $sigmoid$.

$\log \frac{p}{1-p}$ is declared to be equal to $e^{\beta_0 + \beta x}$$\beta_0 + \beta x$ in order to end up with the $sigmoid$.

I think it would have been more straight-forward to say that you want to map a linear function to something between 0 and 1, which you can do with the sigmoid and then you would have been done in one step. It seems a bit weird to introduce the inverse of the sigmoid, claiming that this was some property you want and then solve for $p$, but that might be a matter of taste.

This feels like a bit of a convoluted way to introduce the concept, but alright :D Let me start at a slightly different point.

  1. Maybe in Machine Learning or in other places you have encountered the $sigmoid$ function: $$ sigmoid(S) = \frac{e^S}{1+e^S} = \frac{1}{1+e^{-S}} $$ The sigmoid has the nice property to map any real number $S$ to a number between 0 and 1. This is super when dealing with models that have to represent probabilities.

  2. As in the slides they are looking for a (conditional) probability $p(x)$ they go ahead and implicitly set $$ p(S) = sigmoid(S) $$ They just do it around an extra corner. In order to be consistent with the slides (and to save some tedious writing) I'll now keep talking about the probability $p(x)$ and think of the $sigmoid$-function.

  3. $p(S)$ (i.e. the $sigmoid$) has an inverse function, called the $logit$. Let's try to find the inverse of $p(S)$: $$ p = \frac{1}{1+e^{-S}} $$ and from there we move some stuff around $$ 1+e^{-S} = \frac{1}{p} $$ $$ e^{-S} = \frac{1}{p} - 1 $$ $$ -S = \log \left[\frac{1}{p} - 1 \right] $$ $$ S = \log \left[\frac{1}{\frac{1}{p} - 1} \right] $$ $$ S = \log \left[\frac{p}{1 - p} \right] $$ This is where they start in your notes. We can take any number $S$, punch it into the logit, solve for $p$ and Boom! we've got ourselves a nice conditional probability.

  4. So the next question is: what is $S$? $S$ is supposed to be some function of $x$. There is a wide variety of functions you could use. You could even put a massive Neural Network, but for now, let's stick to linear regression $$ S(x) = \beta_0 + \beta x $$ If you decide to go for $S(x)$ being linear, you can now go step 3. backwards and end up at step 1. with the expression they also show in the notes $$ p(x) = \frac{e^{\beta_0 + \beta x}}{1+e^{\beta_0 + \beta x}} = \frac{1}{1+e^{-(\beta_0 + \beta x)}} $$

So to answer your questions:

$1-p(x)$ is not in the denominator of the distribution, it seems to me they just introduce it in the $\log\frac{p}{1-p}$ in order to end up with the $sigmoid$.

$p(x)$ is not equal to $e^{\beta_0 + \beta x}$, but equal to the $sigmoid$.

$\log \frac{p}{1-p}$ is declared to be equal to $e^{\beta_0 + \beta x}$ in order to end up with the $sigmoid$.

I think it would have been more straight-forward to say that you want to map a linear function to something between 0 and 1, which you can do with the sigmoid and then you would have been done in one step. It seems a bit weird to introduce the inverse of the sigmoid, claiming that this was some property you want and then solve for $p$, but that might be a matter of taste.

This feels like a bit of a convoluted way to introduce the concept, but alright :D Let me start at a slightly different point.

  1. Maybe in Machine Learning or in other places you have encountered the $sigmoid$ function: $$ sigmoid(S) = \frac{e^S}{1+e^S} = \frac{1}{1+e^{-S}} $$ The sigmoid has the nice property to map any real number $S$ to a number between 0 and 1. This is super when dealing with models that have to represent probabilities.

  2. As in the slides they are looking for a (conditional) probability $p(x)$ they go ahead and implicitly set $$ p(S) = sigmoid(S) $$ They just do it around an extra corner. In order to be consistent with the slides (and to save some tedious writing) I'll now keep talking about the probability $p(x)$ and think of the $sigmoid$-function.

  3. $p(S)$ (i.e. the $sigmoid$) has an inverse function, called the $logit$. Let's try to find the inverse of $p(S)$: $$ p = \frac{1}{1+e^{-S}} $$ and from there we move some stuff around $$ 1+e^{-S} = \frac{1}{p} $$ $$ e^{-S} = \frac{1}{p} - 1 $$ $$ -S = \log \left[\frac{1}{p} - 1 \right] $$ $$ S = \log \left[\frac{1}{\frac{1}{p} - 1} \right] $$ $$ S = \log \left[\frac{p}{1 - p} \right] $$ This is where they start in your notes. We can take any number $S$, punch it into the logit, solve for $p$ and Boom! we've got ourselves a nice conditional probability.

  4. So the next question is: what is $S$? $S$ is supposed to be some function of $x$. There is a wide variety of functions you could use. You could even put a massive Neural Network, but for now, let's stick to linear regression $$ S(x) = \beta_0 + \beta x $$ If you decide to go for $S(x)$ being linear, you can now go step 3. backwards and end up at step 1. with the expression they also show in the notes $$ p(x) = \frac{e^{\beta_0 + \beta x}}{1+e^{\beta_0 + \beta x}} = \frac{1}{1+e^{-(\beta_0 + \beta x)}} $$

So to answer your questions:

$1-p(x)$ is not in the denominator of the distribution, it seems to me they just introduce it in the $\log\frac{p}{1-p}$ in order to end up with the $sigmoid$.

$p(x)$ is not equal to $e^{\beta_0 + \beta x}$, but equal to the $sigmoid$.

$\log \frac{p}{1-p}$ is declared to be equal to $\beta_0 + \beta x$ in order to end up with the $sigmoid$.

I think it would have been more straight-forward to say that you want to map a linear function to something between 0 and 1, which you can do with the sigmoid and then you would have been done in one step. It seems a bit weird to introduce the inverse of the sigmoid, claiming that this was some property you want and then solve for $p$, but that might be a matter of taste.

Source Link
matthiaw91
  • 1.6k
  • 6
  • 17

This feels like a bit of a convoluted way to introduce the concept, but alright :D Let me start at a slightly different point.

  1. Maybe in Machine Learning or in other places you have encountered the $sigmoid$ function: $$ sigmoid(S) = \frac{e^S}{1+e^S} = \frac{1}{1+e^{-S}} $$ The sigmoid has the nice property to map any real number $S$ to a number between 0 and 1. This is super when dealing with models that have to represent probabilities.

  2. As in the slides they are looking for a (conditional) probability $p(x)$ they go ahead and implicitly set $$ p(S) = sigmoid(S) $$ They just do it around an extra corner. In order to be consistent with the slides (and to save some tedious writing) I'll now keep talking about the probability $p(x)$ and think of the $sigmoid$-function.

  3. $p(S)$ (i.e. the $sigmoid$) has an inverse function, called the $logit$. Let's try to find the inverse of $p(S)$: $$ p = \frac{1}{1+e^{-S}} $$ and from there we move some stuff around $$ 1+e^{-S} = \frac{1}{p} $$ $$ e^{-S} = \frac{1}{p} - 1 $$ $$ -S = \log \left[\frac{1}{p} - 1 \right] $$ $$ S = \log \left[\frac{1}{\frac{1}{p} - 1} \right] $$ $$ S = \log \left[\frac{p}{1 - p} \right] $$ This is where they start in your notes. We can take any number $S$, punch it into the logit, solve for $p$ and Boom! we've got ourselves a nice conditional probability.

  4. So the next question is: what is $S$? $S$ is supposed to be some function of $x$. There is a wide variety of functions you could use. You could even put a massive Neural Network, but for now, let's stick to linear regression $$ S(x) = \beta_0 + \beta x $$ If you decide to go for $S(x)$ being linear, you can now go step 3. backwards and end up at step 1. with the expression they also show in the notes $$ p(x) = \frac{e^{\beta_0 + \beta x}}{1+e^{\beta_0 + \beta x}} = \frac{1}{1+e^{-(\beta_0 + \beta x)}} $$

So to answer your questions:

$1-p(x)$ is not in the denominator of the distribution, it seems to me they just introduce it in the $\log\frac{p}{1-p}$ in order to end up with the $sigmoid$.

$p(x)$ is not equal to $e^{\beta_0 + \beta x}$, but equal to the $sigmoid$.

$\log \frac{p}{1-p}$ is declared to be equal to $e^{\beta_0 + \beta x}$ in order to end up with the $sigmoid$.

I think it would have been more straight-forward to say that you want to map a linear function to something between 0 and 1, which you can do with the sigmoid and then you would have been done in one step. It seems a bit weird to introduce the inverse of the sigmoid, claiming that this was some property you want and then solve for $p$, but that might be a matter of taste.