1
$\begingroup$

I have been interested in DS and ML recently and logistic regression was on of the first algorithms I learned. In my first course it was said that ln(p/(1-p) was used for the logistic regression. But then I took another course and there was told about sigmoid function for logistic regression. Which of them is better for binary classification? Where to use odds logarithm and where to use sigmoid?

$\endgroup$

1 Answer 1

1
$\begingroup$

A sigmoid function is a function with specific properties, most notably it maps values to the interval $[0,1]$. Often "sigmoid" is used to describe "some" s-shaped function which maps values to the interval $[0,1]$.

If you take the logistic function

$$ f(x) = \frac{L}{ 1 + e^{-k(x-x_0)} },$$

with $k=1, x_0=0, L=1$ ($x_0$ midpoint, $L$ curve max. value, $k$ logistic growth rate), you arrive at:

$$ f(x) = \frac{e^x}{1+e^x}.$$

With (binomial) logistic regression you aim at predicting a probability for some outcome $y$ based on some real valued inputs $x$, so you can employ the logistic function. You can write (omitting indexes $i$):

$$p(y=1) = \frac{exp(x' \beta)}{1+exp(x'\beta)},$$

where $x'\beta = \beta_0 + \beta_1x_1 + ... + \beta_nx_n$.

Now when you look at the odds $\frac{p(y=1)}{1-p(y=1)}$, you can plug in the equation above.

$$ \frac{p}{1-p}=\frac{\frac{exp(\beta_0+x_{1}\beta_1+...+x_{n}\beta_n)}{1+exp(\beta_0+x_{1}\beta_1+...+x_{n}\beta_n)}}{1-\frac{exp(\beta_0+x_{1}\beta_1+...+x_{n}\beta_n)}{1+exp(\beta_0+x_{1}\beta_1+...+x_{n}\beta_n)}}=exp(\beta_0+x_{1}\beta_1+...+x_{n}\beta_n).$$

Now in order to have a linear function (or model) on the right hand side you can take $log$s.

$$\ln\left(\frac{p_i}{1-p_i}\right)=\beta_0+x_{1}\beta_1+...+x_{n}\beta_n.$$

The log-odds (left hand side) are also called logit. The logit serves as a link function between the linear model and the probability.

The R code below looks at a logistic function, odds, and log-odds.

# Logistic function 
x = seq(-10,10,1)
res1 = exp(x) / (exp(x)+1) # Maps some values x to interval [0,1]
plot(x,res1)

p = seq(0,1,0.01) # Probablility that event occurs

# Odds
odds = p/(1-p)
plot(odds,p)

# Logit
logodds = log((p/(1-p))) # Logit maps probabilities to real numbers
plot(logodds,p)
$\endgroup$
2
  • $\begingroup$ Thank you, but in which situations is it better to use each of them? $\endgroup$
    – No Name
    Commented Jan 13, 2022 at 15:37
  • $\begingroup$ logit is the inverse of the logistic function: so logit and logistic function (as "sigmoid"-like function) are two sides of one coin. The logistic regression model is defined as above and finally represented by "logit". $\endgroup$
    – Peter
    Commented Jan 13, 2022 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.