0
$\begingroup$

I've read several articles/tutorials on Logistic Regression and I've come across this idea of log-odds being equal to the weighted sum of features.

i.e. if $p$ is the probability of a sample belonging to positive class(target variable: 1), $(1-p)$ is the probability of it belonging to the negative(target variable: 0). Then, for input features $x_1$, $x_2$, $x_3$ ... $x_n$ and weights $\theta_0, \theta_1, \theta_2... \theta_n$, they write

\begin{equation} log\bigg(\frac{p}{1-p}\bigg) = x_0 \theta_0 + x_1 \theta_1 + x_2 \theta_2 \dots x_n \theta_n \end{equation}

where $x_0$ is bias

Now I get what features are, I get what weighted sum of feature is, I know about the odds, I know log, what I cannot comprehend is, how are those two quantities equal? I've gone through plenty of online articles but they always say that these two are equal, but they don't explain how.

$\endgroup$
0
$\begingroup$

Ordinary regression, Logistic regression, Poisson regression are all examples of Generalized Linear Models (GLM).

A GLM can be broken down into 3 components:

  • A random component: Target Variable $Y_i$ which is assumed to follow a distribution with $E(Y_i)$ and $Var(Y_i)$
  • A systematic component: A linear combination of the features variables known as the linear predictor; $\gamma_i= \beta_{0} + \beta_1{x_1} \dots + \beta_n{x_n}$

  • Link Component: The link function, $g(\cdot)$, connects the random component to the systematic component. If $E(Y_i) = u_{i}$, then the link function connects $\mu_{i}$ to $\gamma_{i}$ through some function $g(\cdot)$, and we then have that $g(\mu_i)= \beta_{0} + \beta_1{x_1} \dots + \beta_n{x_n}$. Moreover, $E(Y_i) = \mu_i = g^{-1}(\beta_{0} + \beta_1{x_1} \dots + \beta_n{x_n})$

The link function for linear regression is the identify link: $g(\mu_i) = \mu_i$

The link function for logistic regression is the logit link: $g(\mu) = log(\frac{\mu}{1-\mu})$. The logit link is used when $\mu$ takes values between 0 and 1 since the function maps to -infinity to infinity.

There are a lot details which I have skimmed over on purpose, GLM is a rich subject. I would recommend reading more about it, here are some references below:

References:

Categorical Data Analysis by Alan Agresti

An Introduction to Categorical Data Analysis

Hope this helps!

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.