2
$\begingroup$

We have two prominent functions (or we can say equations) in logistic regression algorithm: 1. Logistic regression function. 2. Logit function.

  1. I would like to know: a. Which of these equation(s) is/are used in the logistic regression model building process? b. At what stage of model building process which of these equation(s) is/are used?

  2. I know that logit function is used to transform probability values (which range b/w 0 and 1) to real number values (which range b/w -Inf to +Inf). I would like to know the real purpose of logit function in logistic regression modeling process.

Here are few queries which are directly related to the purpose of logit function in Logistic regression modeling:

  1. Has Logit function (i.e. Logit equation LN(P/1-P)) being derived from Logistic Regression equation or its the other way around?

  2. What is the purpose of Logit equation in logistic regression equation? How logit function is used in Logistic regression algorithm? Reason for asking this question will get clear after going through point no. 3 & 4.

  3. Upon building a logistic regression model, we get model coefficients. When we substitute these model coefficients and respective predictor values into the logistic regression equation, we get probability value of being default class (same as the values returned by predict()). a) Does this mean that estimated model coefficient values are determined based on the probability values (computed using logistic regression equation not logit equation) which will be inputed to the likelihood function to determine if it maximizes it or not? If this understanding is correct then, where the logit function is used in the entire process of model building.

  4. Assume that - "Neither logit function is used during model building not during predicting the values". If this is the case then why do we give importance to logit function which is used to map probability values to real number values (ranging between -Inf to +Inf). a) Where exactly the logit function is used in the entire logistic regression model buidling process? Is it while estimating the model coefficients?

  5. The model coefficient estimates that we see upon running summary(lr_model) are determined using linear form of logistic regression equation (logit equation) or the actual logistic regression equation?

It would be great if someone clarifies my doubts.

$\endgroup$
2
$\begingroup$

Has Logit function (i.e. Logit equation LN(P/1-P)) being derived from Logistic Regression equation or its the other way around?

Neither, really. It comes from the probability distribution that you assume in a logistic regression model. First, logistic regression as you probably know is a specific case of a generalized linear model (often abbreviated as "GLM"). A GLM is just that; a generalization of the famous "standard" linear regression model. The "standard" model assumes the conditional distribution of your response, given your predictors (denote this as $Y|X$) is normally distributed. However, in a classification problem we clearly do not have a target variable that can take on any value in the interval $[-\infty, \infty]$, but rather, only two possible outcomes (0 or 1).

A GLM requires three components; a specified probability distribution of $Y|X$ from the exponential family of distributions (important in the context of your question), a linear predictor (i.e. $B_0 + B_1X_1 + B_2X_2 + ...$, denote this as $XB$ in matrix notation), and finally, a specified link function that "links" the linear predictor to the mean (denote this as $\mu$) of the specified probability distribution.

First, let's discuss the probability distribution as this answers the question in quotes. In any logistic regression, you are assuming that $Y|X$ follows a Bernoulli distribution. Recall that a Bernoulli distributed random variable takes on 1 with probability p and 0, with probability 1-p. The mean/expected value of a Bernoulli random variable is therefore $1 *p + 0 * (1-p) = p$. This is clearly much more appropriate to our binary classification problem then say, a normal.

Also recall that I mentioned the importance of the exponential family of distributions. Well, it turns out that all members of this family of distributions have density functions that can be further factored into a very specific form with isolated terms in an exponential. One of these terms is a function that is only an argument of the scale parameter of the distribution. For the Bernoulli distribution, it turns out that this function of the scale parameter is exactly the logit link function, $ln(\frac{p}{1-p})$. Therefore, using the logit link function allows us to map our linear predictor to the exact form of the density function of the Bernoulli distribution.

There are theoretical reasons as to why we in general prefer to use the canonical link function (this is concerned with sufficiency) but there are also many reasons as to why we don't use canonical link functions. For example, the Gamma distribution has support on $[0, \infty]$ but its canonical link function is the negative inverse which runs on $[-\infty, \infty]$. Most prefer to use the log link instead for the Gamma distribution for this reason.

There are other legitimate link functions for classification problems as well. The probit link function, for example, or $ln[-ln(\mu)]$.

What is the purpose of Logit equation in logistic regression equation? How logit function is used in Logistic regression algorithm? Reason for asking this question will get clear after going through point no. 3 & 4.

I basically explained this above. The logit equation, like you correctly state, is used to map the linear predictor $XB$ which runs on $[-\infty, \infty]$ to an interval on $[0,1]$, a probability. That's basically the main purpose of the function but again, there are others you can use that also have supports that run on $[0,1]$, but may have less desirable statistical properties when compared to the logit.

Upon building a logistic regression model, we get model coefficients. When we substitute these model coefficients and respective predictor values into the logistic regression equation, we get probability value of being default class (same as the values returned by predict()). a) Does this mean that estimated model coefficient values are determined based on the probability values (computed using logistic regression equation not logit equation) which will be inputed to the likelihood function to determine if it maximizes it or not? If this understanding is correct then, where the logit function is used in the entire process of model building.

The estimated model coefficient values ARE determined based on what link function you choose. In the fitting process, the link function is what provides the map between the linear predictor with fitted model coefficients and the mean of the fitted distribution (in our case for logistic regression, the parameter $p$ as explained above). The objective function = log likelihood will end up looking like this:

$$\sum_{i=1}^{n} (y_i * ln(p_i) + (1-y_i) * ln(1-p_i))$$

where $y_i =$ 0 or 1, $p_i =$ predicted probability of class 1 for the ith observation, and $n =$ number of observations.

Notice how there is no linear predictor and no fitted coefficients in this expression. But this is where the link function comes into play; namely,

$X_iB = ln(\frac{p_i}{1-p_i}) \rightarrow p_i = \frac{exp(X_iB)}{1 + exp(X_iB)}$. So the fitted coefficients in the linear predictor are influenced by your choice of link function. There is no way to relate the linear predictor and its coefficients to the loglikelihood function otherwise. We find the most optimal $B$ that maximize the expression above.

Assume that - "Neither logit function is used during model building not during predicting the values". If this is the case then why do we give importance to logit function which is used to map probability values to real number values (ranging between -Inf to +Inf). a) Where exactly the logit function is used in the entire logistic regression model buidling process? Is it while estimating the model coefficients?

See the above to (hopefully) be convinced that the logit link function is used during model fitting and is not arbitrary. This isn't a very rigorous explanation but I hope it clears up some of your doubts.

The model coefficient estimates that we see upon running summary(lr_model) are determined using linear form of logistic regression equation (logit equation) or the actual logistic regression equation?

In general, you can typically specify whether you want just the linear predictor (i.e. $X_iB$) or the predicted probabilities (i.e. $p_i = \frac{exp(X_iB)}{1+exp(X_iB)}$). In R this is done via. an argument to the predict function, not sure how it is done in Python but I assume it is similar.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Logit function is typically used as a "trick" in order to run logistic regressions. Logistic regression is based on:

y = Sigmoid( X )

that is: you transform your regression equation using the Sigmoid function. However, it is much simpler, for the computer, tu run the very same model by reverting back the Sigmoid transformation. You get:

logit( y ) = X

Logit is thus the inverse of Sigmoid. Thanks to this trick, you computer is tricked to run a logistic regression, while thinking it is a simple linear regression. The only difference is that you are not running a regression on y anymore, but on its logit(), that is: on the natural log of its odds ratio.

Hope this helps, otherwise let me know.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.